Creating new pipeline using seurat v4.0.2 available 2021.06.23
Important notes:

Load libraries required for Seuratv4

knitr::opts_knit$set(root.dir = "~/Desktop/10XGenomicsData/msAggr_scRNASeq/")
library(dplyr)
library(Seurat)
library(patchwork)
library(ggplot2)
# library(clustree)

store session info

sink("msAggr_seurat-v1.20210623")
sessionInfo()
sink()

A note about using SCTransform versus ScaleData

https://bioconductor.org/packages/3.10/workflows/vignettes/simpleSingleCell/inst/doc/batch.html#62_for_gene-based_analyses >You can also normalize and scale data for the RNA assay. There are numerous resources on this, but Aaron Lun describes why the original log-normalized values should be used for DE and visualizations of expression quite well here: > >For gene-based procedures like differential expression (DE) analyses or gene network construction, it is desirable to use the original log-expression values or counts. The corrected values are only used to obtain cell-level results such as clusters or trajectories. Batch effects are handled explicitly using blocking terms or via a meta-analysis across batches. We do not use the corrected values directly in gene-based analyses, for various reasons: > >It is usually inappropriate to perform DE analyses on batch-corrected values, due to the failure to model the uncertainty of the correction. This usually results in loss of type I error control, i.e., more false positives than expected. > >The correction does not preserve the mean-variance relationship. Applications of common DE methods like edgeR or limma are unlikely to be valid. > >Batch correction may (correctly) remove biological differences between batches in the course of mapping all cells onto a common coordinate system. Returning to the uncorrected expression values provides an opportunity for detecting such differences if they are of interest. Conversely, if the batch correction made a mistake, the use of the uncorrected expression values provides an important sanity check. > >In addition, the normalized values in SCT and integrated assays don’t necessary correspond to per-gene expression values anyway, rather containing residuals (in the case of the scale.data slot for each).

Mess with how to load 4 cell populations into single seurat object

SET SEED?????!!!!!

Set global variables

projectName <- "msAggr"
jackstraw.dim <- 40
source("msAggr_AnalysisCode/read_10XGenomics_data.R")
source("msAggr_AnalysisCode/PercentVariance.R")
setwd("../cellRanger/") # temporarily changing wd only works if you run the entire chunk at once
Warning: The working directory was changed to /Users/heustonef/Desktop/10XGenomicsData/cellRanger inside a notebook chunk. The working directory will be reset when the chunk is finished running. Use the knitr root.dir option in the setup chunk to change the working directory for notebook chunks.
data_file.list <- read_10XGenomics_data(sample.list = c("LSKm2", "CMPm2", "MEPm", "GMPm"))
data.object<-Read10X(data_file.list)
seurat.object<- CreateSeuratObject(counts = data.object, min.cells = 3, min.genes = 200, project = projectName)

Clean up to free memory

remove(data.object)

Add mitochondrial metadata and plot some basic features

seurat.object[["percent.mt"]] <- PercentageFeatureSet(seurat.object, pattern = "^mt-")
VlnPlot(seurat.object, features = c("nFeature_RNA", "nCount_RNA", "percent.mt"), ncol = 3, pt.size = 0, fill.by = 'orig.ident', )

plot1 <- FeatureScatter(seurat.object, feature1 = "nCount_RNA", feature2 = "percent.mt", group.by = "orig.ident", pt.size = 0.01)
plot2 <- FeatureScatter(seurat.object, feature1 = "nCount_RNA", feature2 = "nFeature_RNA", group.by = "orig.ident", pt.size = 0.01)
plot1 + plot2

remove low quality cells require: nFeature_RNA between 200 and 4000 (inclusive) require: percent.mt <= 5

print(paste("original object:", nrow(seurat.object@meta.data), "cells", sep = " "))
[1] "original object: 41950 cells"
seurat.object <- subset(seurat.object, 
                                                subset = nFeature_RNA >=200 & 
                                                    nFeature_RNA <= 4000 & 
                                                    percent.mt <= 5
                                                )
print(paste("new object:", nrow(seurat.object@meta.data), "cells", sep = " "))
[1] "new object: 37104 cells"

Normalization

Struggling to wrap my head around this one. It seems that SCTransform is best for batch correction, but NormalizeData and ScaleData are best for DGE. Several vignettes have performed both

`selection.method
How to choose top variable features. Choose one of :

vst: First, fits a line to the relationship of log(variance) and log(mean) using local polynomial regression (loess). Then standardizes the feature values using the observed mean and expected variance (given by the fitted line). Feature variance is then calculated on the standardized values after clipping to a maximum (see clip.max parameter).

mean.var.plot (mvp): First, uses a function to calculate average expression (mean.function) and dispersion (dispersion.function) for each feature. Next, divides features into num.bin (deafult 20) bins based on their average expression, and calculates z-scores for dispersion within each bin. The purpose of this is to identify variable features while controlling for the strong relationship between variability and average expression.

dispersion (disp): selects the genes with the highest dispersion values`

seurat.object <- NormalizeData(seurat.object, normalization.method = "LogNormalize", scale.factor = 10000)
Performing log-normalization
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|

Find variable features

seurat.object <- FindVariableFeatures(seurat.object, selection.method = "vst", nfeatures = 2000)
Calculating gene variances
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Calculating feature variances of standardized and clipped values
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
top10 <- head(VariableFeatures(seurat.object), 10)
plot1 <- VariableFeaturePlot(seurat.object)
plot2 <- LabelPoints(plot = plot1, points = top10, repel = TRUE)
When using repel, set xnudge and ynudge to 0 for optimal results
plot1 + plot2
Warning: Transformation introduced infinite values in continuous x-axis
Warning: Removed 2 rows containing missing values (geom_point).
Warning: Transformation introduced infinite values in continuous x-axis
Warning: Removed 2 rows containing missing values (geom_point).

Scale data (linear transformation)

all.genes <- rownames(seurat.object)
seurat.object <- ScaleData(seurat.object, features = all.genes, vars.to.regress = c("nCount_RNA", "nFeature_RNA"))
Regressing out nCount_RNA, nFeature_RNA

  |                                                                                                                                                                                        
  |                                                                                                                                                                                  |   0%
  |                                                                                                                                                                                        
  |=                                                                                                                                                                                 |   0%
  |                                                                                                                                                                                        
  |=                                                                                                                                                                                 |   1%
  |                                                                                                                                                                                        
  |==                                                                                                                                                                                |   1%
  |                                                                                                                                                                                        
  |===                                                                                                                                                                               |   1%
  |                                                                                                                                                                                        
  |===                                                                                                                                                                               |   2%
  |                                                                                                                                                                                        
  |====                                                                                                                                                                              |   2%
  |                                                                                                                                                                                        
  |====                                                                                                                                                                              |   3%
  |                                                                                                                                                                                        
  |=====                                                                                                                                                                             |   3%
  |                                                                                                                                                                                        
  |======                                                                                                                                                                            |   3%
  |                                                                                                                                                                                        
  |======                                                                                                                                                                            |   4%
  |                                                                                                                                                                                        
  |=======                                                                                                                                                                           |   4%
  |                                                                                                                                                                                        
  |========                                                                                                                                                                          |   4%
  |                                                                                                                                                                                        
  |========                                                                                                                                                                          |   5%
  |                                                                                                                                                                                        
  |=========                                                                                                                                                                         |   5%
  |                                                                                                                                                                                        
  |==========                                                                                                                                                                        |   5%
  |                                                                                                                                                                                        
  |==========                                                                                                                                                                        |   6%
  |                                                                                                                                                                                        
  |===========                                                                                                                                                                       |   6%
  |                                                                                                                                                                                        
  |============                                                                                                                                                                      |   6%
  |                                                                                                                                                                                        
  |============                                                                                                                                                                      |   7%
  |                                                                                                                                                                                        
  |=============                                                                                                                                                                     |   7%
  |                                                                                                                                                                                        
  |=============                                                                                                                                                                     |   8%
  |                                                                                                                                                                                        
  |==============                                                                                                                                                                    |   8%
  |                                                                                                                                                                                        
  |===============                                                                                                                                                                   |   8%
  |                                                                                                                                                                                        
  |===============                                                                                                                                                                   |   9%
  |                                                                                                                                                                                        
  |================                                                                                                                                                                  |   9%
  |                                                                                                                                                                                        
  |=================                                                                                                                                                                 |   9%
  |                                                                                                                                                                                        
  |=================                                                                                                                                                                 |  10%
  |                                                                                                                                                                                        
  |==================                                                                                                                                                                |  10%
  |                                                                                                                                                                                        
  |===================                                                                                                                                                               |  10%
  |                                                                                                                                                                                        
  |===================                                                                                                                                                               |  11%
  |                                                                                                                                                                                        
  |====================                                                                                                                                                              |  11%
  |                                                                                                                                                                                        
  |====================                                                                                                                                                              |  12%
  |                                                                                                                                                                                        
  |=====================                                                                                                                                                             |  12%
  |                                                                                                                                                                                        
  |======================                                                                                                                                                            |  12%
  |                                                                                                                                                                                        
  |======================                                                                                                                                                            |  13%
  |                                                                                                                                                                                        
  |=======================                                                                                                                                                           |  13%
  |                                                                                                                                                                                        
  |========================                                                                                                                                                          |  13%
  |                                                                                                                                                                                        
  |========================                                                                                                                                                          |  14%
  |                                                                                                                                                                                        
  |=========================                                                                                                                                                         |  14%
  |                                                                                                                                                                                        
  |==========================                                                                                                                                                        |  14%
  |                                                                                                                                                                                        
  |==========================                                                                                                                                                        |  15%
  |                                                                                                                                                                                        
  |===========================                                                                                                                                                       |  15%
  |                                                                                                                                                                                        
  |============================                                                                                                                                                      |  15%
  |                                                                                                                                                                                        
  |============================                                                                                                                                                      |  16%
  |                                                                                                                                                                                        
  |=============================                                                                                                                                                     |  16%
  |                                                                                                                                                                                        
  |=============================                                                                                                                                                     |  17%
  |                                                                                                                                                                                        
  |==============================                                                                                                                                                    |  17%
  |                                                                                                                                                                                        
  |===============================                                                                                                                                                   |  17%
  |                                                                                                                                                                                        
  |===============================                                                                                                                                                   |  18%
  |                                                                                                                                                                                        
  |================================                                                                                                                                                  |  18%
  |                                                                                                                                                                                        
  |=================================                                                                                                                                                 |  18%
  |                                                                                                                                                                                        
  |=================================                                                                                                                                                 |  19%
  |                                                                                                                                                                                        
  |==================================                                                                                                                                                |  19%
  |                                                                                                                                                                                        
  |===================================                                                                                                                                               |  19%
  |                                                                                                                                                                                        
  |===================================                                                                                                                                               |  20%
  |                                                                                                                                                                                        
  |====================================                                                                                                                                              |  20%
  |                                                                                                                                                                                        
  |====================================                                                                                                                                              |  21%
  |                                                                                                                                                                                        
  |=====================================                                                                                                                                             |  21%
  |                                                                                                                                                                                        
  |======================================                                                                                                                                            |  21%
  |                                                                                                                                                                                        
  |======================================                                                                                                                                            |  22%
  |                                                                                                                                                                                        
  |=======================================                                                                                                                                           |  22%
  |                                                                                                                                                                                        
  |========================================                                                                                                                                          |  22%
  |                                                                                                                                                                                        
  |========================================                                                                                                                                          |  23%
  |                                                                                                                                                                                        
  |=========================================                                                                                                                                         |  23%
  |                                                                                                                                                                                        
  |==========================================                                                                                                                                        |  23%
  |                                                                                                                                                                                        
  |==========================================                                                                                                                                        |  24%
  |                                                                                                                                                                                        
  |===========================================                                                                                                                                       |  24%
  |                                                                                                                                                                                        
  |============================================                                                                                                                                      |  24%
  |                                                                                                                                                                                        
  |============================================                                                                                                                                      |  25%
  |                                                                                                                                                                                        
  |=============================================                                                                                                                                     |  25%
  |                                                                                                                                                                                        
  |=============================================                                                                                                                                     |  26%
  |                                                                                                                                                                                        
  |==============================================                                                                                                                                    |  26%
  |                                                                                                                                                                                        
  |===============================================                                                                                                                                   |  26%
  |                                                                                                                                                                                        
  |===============================================                                                                                                                                   |  27%
  |                                                                                                                                                                                        
  |================================================                                                                                                                                  |  27%
  |                                                                                                                                                                                        
  |=================================================                                                                                                                                 |  27%
  |                                                                                                                                                                                        
  |=================================================                                                                                                                                 |  28%
  |                                                                                                                                                                                        
  |==================================================                                                                                                                                |  28%
  |                                                                                                                                                                                        
  |===================================================                                                                                                                               |  28%
  |                                                                                                                                                                                        
  |===================================================                                                                                                                               |  29%
  |                                                                                                                                                                                        
  |====================================================                                                                                                                              |  29%
  |                                                                                                                                                                                        
  |=====================================================                                                                                                                             |  29%
  |                                                                                                                                                                                        
  |=====================================================                                                                                                                             |  30%
  |                                                                                                                                                                                        
  |======================================================                                                                                                                            |  30%
  |                                                                                                                                                                                        
  |======================================================                                                                                                                            |  31%
  |                                                                                                                                                                                        
  |=======================================================                                                                                                                           |  31%
  |                                                                                                                                                                                        
  |========================================================                                                                                                                          |  31%
  |                                                                                                                                                                                        
  |========================================================                                                                                                                          |  32%
  |                                                                                                                                                                                        
  |=========================================================                                                                                                                         |  32%
  |                                                                                                                                                                                        
  |==========================================================                                                                                                                        |  32%
  |                                                                                                                                                                                        
  |==========================================================                                                                                                                        |  33%
  |                                                                                                                                                                                        
  |===========================================================                                                                                                                       |  33%
  |                                                                                                                                                                                        
  |============================================================                                                                                                                      |  33%
  |                                                                                                                                                                                        
  |============================================================                                                                                                                      |  34%
  |                                                                                                                                                                                        
  |=============================================================                                                                                                                     |  34%
  |                                                                                                                                                                                        
  |=============================================================                                                                                                                     |  35%
  |                                                                                                                                                                                        
  |==============================================================                                                                                                                    |  35%
  |                                                                                                                                                                                        
  |===============================================================                                                                                                                   |  35%
  |                                                                                                                                                                                        
  |===============================================================                                                                                                                   |  36%
  |                                                                                                                                                                                        
  |================================================================                                                                                                                  |  36%
  |                                                                                                                                                                                        
  |=================================================================                                                                                                                 |  36%
  |                                                                                                                                                                                        
  |=================================================================                                                                                                                 |  37%
  |                                                                                                                                                                                        
  |==================================================================                                                                                                                |  37%
  |                                                                                                                                                                                        
  |===================================================================                                                                                                               |  37%
  |                                                                                                                                                                                        
  |===================================================================                                                                                                               |  38%
  |                                                                                                                                                                                        
  |====================================================================                                                                                                              |  38%
  |                                                                                                                                                                                        
  |=====================================================================                                                                                                             |  38%
  |                                                                                                                                                                                        
  |=====================================================================                                                                                                             |  39%
  |                                                                                                                                                                                        
  |======================================================================                                                                                                            |  39%
  |                                                                                                                                                                                        
  |======================================================================                                                                                                            |  40%
  |                                                                                                                                                                                        
  |=======================================================================                                                                                                           |  40%
  |                                                                                                                                                                                        
  |========================================================================                                                                                                          |  40%
  |                                                                                                                                                                                        
  |========================================================================                                                                                                          |  41%Warning in doTryCatch(return(expr), name, parentenv, handler) :
  restarting interrupted promise evaluation

  |                                                                                                                                                                                        
  |=========================================================================                                                                                                         |  41%
  |                                                                                                                                                                                        
  |==========================================================================                                                                                                        |  41%
  |                                                                                                                                                                                        
  |==========================================================================                                                                                                        |  42%
  |                                                                                                                                                                                        
  |===========================================================================                                                                                                       |  42%
  |                                                                                                                                                                                        
  |============================================================================                                                                                                      |  42%
  |                                                                                                                                                                                        
  |============================================================================                                                                                                      |  43%
  |                                                                                                                                                                                        
  |=============================================================================                                                                                                     |  43%
  |                                                                                                                                                                                        
  |=============================================================================                                                                                                     |  44%
  |                                                                                                                                                                                        
  |==============================================================================                                                                                                    |  44%
  |                                                                                                                                                                                        
  |===============================================================================                                                                                                   |  44%
  |                                                                                                                                                                                        
  |===============================================================================                                                                                                   |  45%
  |                                                                                                                                                                                        
  |================================================================================                                                                                                  |  45%
  |                                                                                                                                                                                        
  |=================================================================================                                                                                                 |  45%
  |                                                                                                                                                                                        
  |=================================================================================                                                                                                 |  46%
  |                                                                                                                                                                                        
  |==================================================================================                                                                                                |  46%
  |                                                                                                                                                                                        
  |===================================================================================                                                                                               |  46%
  |                                                                                                                                                                                        
  |===================================================================================                                                                                               |  47%
  |                                                                                                                                                                                        
  |====================================================================================                                                                                              |  47%
  |                                                                                                                                                                                        
  |=====================================================================================                                                                                             |  47%
  |                                                                                                                                                                                        
  |=====================================================================================                                                                                             |  48%
  |                                                                                                                                                                                        
  |======================================================================================                                                                                            |  48%
  |                                                                                                                                                                                        
  |======================================================================================                                                                                            |  49%
  |                                                                                                                                                                                        
  |=======================================================================================                                                                                           |  49%
  |                                                                                                                                                                                        
  |========================================================================================                                                                                          |  49%
  |                                                                                                                                                                                        
  |========================================================================================                                                                                          |  50%
  |                                                                                                                                                                                        
  |=========================================================================================                                                                                         |  50%
  |                                                                                                                                                                                        
  |==========================================================================================                                                                                        |  50%
  |                                                                                                                                                                                        
  |==========================================================================================                                                                                        |  51%
  |                                                                                                                                                                                        
  |===========================================================================================                                                                                       |  51%
  |                                                                                                                                                                                        
  |============================================================================================                                                                                      |  51%
  |                                                                                                                                                                                        
  |============================================================================================                                                                                      |  52%
  |                                                                                                                                                                                        
  |=============================================================================================                                                                                     |  52%
  |                                                                                                                                                                                        
  |=============================================================================================                                                                                     |  53%
  |                                                                                                                                                                                        
  |==============================================================================================                                                                                    |  53%
  |                                                                                                                                                                                        
  |===============================================================================================                                                                                   |  53%
  |                                                                                                                                                                                        
  |===============================================================================================                                                                                   |  54%
  |                                                                                                                                                                                        
  |================================================================================================                                                                                  |  54%
  |                                                                                                                                                                                        
  |=================================================================================================                                                                                 |  54%
  |                                                                                                                                                                                        
  |=================================================================================================                                                                                 |  55%
  |                                                                                                                                                                                        
  |==================================================================================================                                                                                |  55%
  |                                                                                                                                                                                        
  |===================================================================================================                                                                               |  55%
  |                                                                                                                                                                                        
  |===================================================================================================                                                                               |  56%
  |                                                                                                                                                                                        
  |====================================================================================================                                                                              |  56%
  |                                                                                                                                                                                        
  |=====================================================================================================                                                                             |  56%
  |                                                                                                                                                                                        
  |=====================================================================================================                                                                             |  57%
  |                                                                                                                                                                                        
  |======================================================================================================                                                                            |  57%
  |                                                                                                                                                                                        
  |======================================================================================================                                                                            |  58%
  |                                                                                                                                                                                        
  |=======================================================================================================                                                                           |  58%
  |                                                                                                                                                                                        
  |========================================================================================================                                                                          |  58%
  |                                                                                                                                                                                        
  |========================================================================================================                                                                          |  59%
  |                                                                                                                                                                                        
  |=========================================================================================================                                                                         |  59%
  |                                                                                                                                                                                        
  |==========================================================================================================                                                                        |  59%
  |                                                                                                                                                                                        
  |==========================================================================================================                                                                        |  60%
  |                                                                                                                                                                                        
  |===========================================================================================================                                                                       |  60%
  |                                                                                                                                                                                        
  |============================================================================================================                                                                      |  60%
  |                                                                                                                                                                                        
  |============================================================================================================                                                                      |  61%
  |                                                                                                                                                                                        
  |=============================================================================================================                                                                     |  61%
  |                                                                                                                                                                                        
  |=============================================================================================================                                                                     |  62%
  |                                                                                                                                                                                        
  |==============================================================================================================                                                                    |  62%
  |                                                                                                                                                                                        
  |===============================================================================================================                                                                   |  62%
  |                                                                                                                                                                                        
  |===============================================================================================================                                                                   |  63%
  |                                                                                                                                                                                        
  |================================================================================================================                                                                  |  63%
  |                                                                                                                                                                                        
  |=================================================================================================================                                                                 |  63%
  |                                                                                                                                                                                        
  |=================================================================================================================                                                                 |  64%
  |                                                                                                                                                                                        
  |==================================================================================================================                                                                |  64%
  |                                                                                                                                                                                        
  |===================================================================================================================                                                               |  64%
  |                                                                                                                                                                                        
  |===================================================================================================================                                                               |  65%
  |                                                                                                                                                                                        
  |====================================================================================================================                                                              |  65%
  |                                                                                                                                                                                        
  |=====================================================================================================================                                                             |  65%
  |                                                                                                                                                                                        
  |=====================================================================================================================                                                             |  66%
  |                                                                                                                                                                                        
  |======================================================================================================================                                                            |  66%
  |                                                                                                                                                                                        
  |======================================================================================================================                                                            |  67%
  |                                                                                                                                                                                        
  |=======================================================================================================================                                                           |  67%
  |                                                                                                                                                                                        
  |========================================================================================================================                                                          |  67%
  |                                                                                                                                                                                        
  |========================================================================================================================                                                          |  68%
  |                                                                                                                                                                                        
  |=========================================================================================================================                                                         |  68%
  |                                                                                                                                                                                        
  |==========================================================================================================================                                                        |  68%
  |                                                                                                                                                                                        
  |==========================================================================================================================                                                        |  69%
  |                                                                                                                                                                                        
  |===========================================================================================================================                                                       |  69%
  |                                                                                                                                                                                        
  |============================================================================================================================                                                      |  69%
  |                                                                                                                                                                                        
  |============================================================================================================================                                                      |  70%
  |                                                                                                                                                                                        
  |=============================================================================================================================                                                     |  70%
  |                                                                                                                                                                                        
  |=============================================================================================================================                                                     |  71%
  |                                                                                                                                                                                        
  |==============================================================================================================================                                                    |  71%
  |                                                                                                                                                                                        
  |===============================================================================================================================                                                   |  71%
  |                                                                                                                                                                                        
  |===============================================================================================================================                                                   |  72%
  |                                                                                                                                                                                        
  |================================================================================================================================                                                  |  72%
  |                                                                                                                                                                                        
  |=================================================================================================================================                                                 |  72%
  |                                                                                                                                                                                        
  |=================================================================================================================================                                                 |  73%
  |                                                                                                                                                                                        
  |==================================================================================================================================                                                |  73%
  |                                                                                                                                                                                        
  |===================================================================================================================================                                               |  73%
  |                                                                                                                                                                                        
  |===================================================================================================================================                                               |  74%
  |                                                                                                                                                                                        
  |====================================================================================================================================                                              |  74%
  |                                                                                                                                                                                        
  |=====================================================================================================================================                                             |  74%
  |                                                                                                                                                                                        
  |=====================================================================================================================================                                             |  75%
  |                                                                                                                                                                                        
  |======================================================================================================================================                                            |  75%
  |                                                                                                                                                                                        
  |======================================================================================================================================                                            |  76%
  |                                                                                                                                                                                        
  |=======================================================================================================================================                                           |  76%
  |                                                                                                                                                                                        
  |========================================================================================================================================                                          |  76%
  |                                                                                                                                                                                        
  |========================================================================================================================================                                          |  77%
  |                                                                                                                                                                                        
  |=========================================================================================================================================                                         |  77%
  |                                                                                                                                                                                        
  |==========================================================================================================================================                                        |  77%
  |                                                                                                                                                                                        
  |==========================================================================================================================================                                        |  78%
  |                                                                                                                                                                                        
  |===========================================================================================================================================                                       |  78%
  |                                                                                                                                                                                        
  |============================================================================================================================================                                      |  78%
  |                                                                                                                                                                                        
  |============================================================================================================================================                                      |  79%
  |                                                                                                                                                                                        
  |=============================================================================================================================================                                     |  79%
  |                                                                                                                                                                                        
  |==============================================================================================================================================                                    |  79%
  |                                                                                                                                                                                        
  |==============================================================================================================================================                                    |  80%
  |                                                                                                                                                                                        
  |===============================================================================================================================================                                   |  80%
  |                                                                                                                                                                                        
  |===============================================================================================================================================                                   |  81%
  |                                                                                                                                                                                        
  |================================================================================================================================================                                  |  81%
  |                                                                                                                                                                                        
  |=================================================================================================================================================                                 |  81%
  |                                                                                                                                                                                        
  |=================================================================================================================================================                                 |  82%
  |                                                                                                                                                                                        
  |==================================================================================================================================================                                |  82%
  |                                                                                                                                                                                        
  |===================================================================================================================================================                               |  82%
  |                                                                                                                                                                                        
  |===================================================================================================================================================                               |  83%
  |                                                                                                                                                                                        
  |====================================================================================================================================================                              |  83%
  |                                                                                                                                                                                        
  |=====================================================================================================================================================                             |  83%
  |                                                                                                                                                                                        
  |=====================================================================================================================================================                             |  84%
  |                                                                                                                                                                                        
  |======================================================================================================================================================                            |  84%
  |                                                                                                                                                                                        
  |======================================================================================================================================================                            |  85%
  |                                                                                                                                                                                        
  |=======================================================================================================================================================                           |  85%
  |                                                                                                                                                                                        
  |========================================================================================================================================================                          |  85%
  |                                                                                                                                                                                        
  |========================================================================================================================================================                          |  86%
  |                                                                                                                                                                                        
  |=========================================================================================================================================================                         |  86%
  |                                                                                                                                                                                        
  |==========================================================================================================================================================                        |  86%
  |                                                                                                                                                                                        
  |==========================================================================================================================================================                        |  87%
  |                                                                                                                                                                                        
  |===========================================================================================================================================================                       |  87%
  |                                                                                                                                                                                        
  |============================================================================================================================================================                      |  87%
  |                                                                                                                                                                                        
  |============================================================================================================================================================                      |  88%
  |                                                                                                                                                                                        
  |=============================================================================================================================================================                     |  88%
  |                                                                                                                                                                                        
  |==============================================================================================================================================================                    |  88%
  |                                                                                                                                                                                        
  |==============================================================================================================================================================                    |  89%
  |                                                                                                                                                                                        
  |===============================================================================================================================================================                   |  89%
  |                                                                                                                                                                                        
  |===============================================================================================================================================================                   |  90%
  |                                                                                                                                                                                        
  |================================================================================================================================================================                  |  90%
  |                                                                                                                                                                                        
  |=================================================================================================================================================================                 |  90%
  |                                                                                                                                                                                        
  |=================================================================================================================================================================                 |  91%
  |                                                                                                                                                                                        
  |==================================================================================================================================================================                |  91%
  |                                                                                                                                                                                        
  |===================================================================================================================================================================               |  91%
  |                                                                                                                                                                                        
  |===================================================================================================================================================================               |  92%
  |                                                                                                                                                                                        
  |====================================================================================================================================================================              |  92%
  |                                                                                                                                                                                        
  |=====================================================================================================================================================================             |  92%
  |                                                                                                                                                                                        
  |=====================================================================================================================================================================             |  93%
  |                                                                                                                                                                                        
  |======================================================================================================================================================================            |  93%
  |                                                                                                                                                                                        
  |======================================================================================================================================================================            |  94%
  |                                                                                                                                                                                        
  |=======================================================================================================================================================================           |  94%
  |                                                                                                                                                                                        
  |========================================================================================================================================================================          |  94%
  |                                                                                                                                                                                        
  |========================================================================================================================================================================          |  95%
  |                                                                                                                                                                                        
  |=========================================================================================================================================================================         |  95%
  |                                                                                                                                                                                        
  |==========================================================================================================================================================================        |  95%
  |                                                                                                                                                                                        
  |==========================================================================================================================================================================        |  96%
  |                                                                                                                                                                                        
  |===========================================================================================================================================================================       |  96%
  |                                                                                                                                                                                        
  |============================================================================================================================================================================      |  96%
  |                                                                                                                                                                                        
  |============================================================================================================================================================================      |  97%
  |                                                                                                                                                                                        
  |=============================================================================================================================================================================     |  97%
  |                                                                                                                                                                                        
  |==============================================================================================================================================================================    |  97%
  |                                                                                                                                                                                        
  |==============================================================================================================================================================================    |  98%
  |                                                                                                                                                                                        
  |===============================================================================================================================================================================   |  98%
  |                                                                                                                                                                                        
  |===============================================================================================================================================================================   |  99%
  |                                                                                                                                                                                        
  |================================================================================================================================================================================  |  99%
  |                                                                                                                                                                                        
  |================================================================================================================================================================================= |  99%
  |                                                                                                                                                                                        
  |================================================================================================================================================================================= | 100%
  |                                                                                                                                                                                        
  |==================================================================================================================================================================================| 100%
Centering and scaling data matrix

  |                                                                                                                                                                                                  
  |                                                                                                                                                                                            |   0%
  |                                                                                                                                                                                                  
  |==========                                                                                                                                                                                  |   5%
  |                                                                                                                                                                                                  
  |====================                                                                                                                                                                        |  11%
  |                                                                                                                                                                                                  
  |==============================                                                                                                                                                              |  16%
  |                                                                                                                                                                                                  
  |========================================                                                                                                                                                    |  21%
  |                                                                                                                                                                                                  
  |=================================================                                                                                                                                           |  26%
  |                                                                                                                                                                                                  
  |===========================================================                                                                                                                                 |  32%
  |                                                                                                                                                                                                  
  |=====================================================================                                                                                                                       |  37%
  |                                                                                                                                                                                                  
  |===============================================================================                                                                                                             |  42%
  |                                                                                                                                                                                                  
  |=========================================================================================                                                                                                   |  47%
  |                                                                                                                                                                                                  
  |===================================================================================================                                                                                         |  53%
  |                                                                                                                                                                                                  
  |=============================================================================================================                                                                               |  58%
  |                                                                                                                                                                                                  
  |=======================================================================================================================                                                                     |  63%
  |                                                                                                                                                                                                  
  |=================================================================================================================================                                                           |  68%
  |                                                                                                                                                                                                  
  |===========================================================================================================================================                                                 |  74%
  |                                                                                                                                                                                                  
  |====================================================================================================================================================                                        |  79%
  |                                                                                                                                                                                                  
  |==============================================================================================================================================================                              |  84%
  |                                                                                                                                                                                                  
  |========================================================================================================================================================================                    |  89%
  |                                                                                                                                                                                                  
  |==================================================================================================================================================================================          |  95%
  |                                                                                                                                                                                                  
  |============================================================================================================================================================================================| 100%

Save progress

save.image(file = paste0(projectName, '.RData'))

PCA

linear dimensional reduction. Default are based on VariableFeatures, but can be changed

seurat.object <- RunPCA(seurat.object, features = VariableFeatures(object = seurat.object))
PC_ 1 
Positive:  Car2, Car1, Blvrb, Klf1, Mt2, Vamp5, Ermap, Aqp1, C1qtnf12, Rhd 
       Sphk1, Ces2g, Tspo2, Cldn13, Gm15915, Slc38a5, Gstm5, Smim1, Abcb4, Gclm 
       Ybx3, Nxpe2, Ctse, Cenpv, Stard10, Mns1, Gata1, Slc25a21, Sdsl, Epor 
Negative:  Tmsb4x, Prtn3, Mpo, Ctsg, Tyrobp, Plac8, Elane, Clec12a, Ly6c2, Slpi 
       Sh3bgrl3, Anxa3, Pkm, Hp, Emb, Fcer1g, Ms4a3, Pgam1, Ms4a6c, Irf8 
       Coro1a, Serpinb1a, BC035044, Lgals1, H2afy, Arhgdib, Igsf6, Alas1, Ly86, Cst3 
PC_ 2 
Positive:  Ctla2a, Malat1, Ifitm1, Hlf, Gimap6, Jund, Fos, Tsc22d1, Ltb, Gimap1 
       Tmem176b, Pim1, Adgrl4, Ifitm3, Sox4, Zfp36, Adgrg1, Cd34, Gcnt2, Rgs1 
       Klf2, Gm5111, Cd27, Ypel3, Klf6, Gm19590, Dusp1, Dusp2, Ifi203, Shisa5 
Negative:  H2afz, Ybx1, Ppia, Atp5g1, Ran, Mt1, Ranbp1, Cycs, Nme1, Hmgb2 
       Cks2, Slc25a5, Cox5a, Cks1b, Ap3s1, Atpif1, Dut, Eif5a, Ptma, Sdf2l1 
       Nhp2, Dynll1, Elane, Mrpl18, Atp5o, Ms4a3, Tmem14c, Chchd2, Ly6c2, Birc5 
PC_ 3 
Positive:  Ube2c, Nusap1, Plbd1, Cenpf, Birc5, Lgals3, Aif1, Mki67, Id2, Hmmr 
       Top2a, Prc1, Ifi205, Cdca8, Kif23, Batf3, H2afx, Cenpa, Tpx2, Ccnb1 
       Ccnb2, Kif22, Cdc20, Plk1, Cenpe, Hmgb2, Pimreg, Racgap1, H2-Aa, Cdca3 
Negative:  Cd63, Srgn, Mif, Ung, Prtn3, Rgcc, Ms4a3, Srm, Gstm1, Elane 
       Nkg7, Hspd1, Cebpe, Alas1, Fkbp11, Hsp90ab1, Mpo, Ctsg, Prdx6, Trem3 
       Calr, Gsr, Serpinb1a, Rps2, Nme1, Prss57, Anxa3, Cst7, Fcgr3, Rack1 
PC_ 4 
Positive:  Srgn, Nkg7, Itga2b, Cd9, Pf4, Ube2c, Lockd, Apoe, Cavin2, Nusap1 
       Cenpf, H1fx, Cks2, Serpine2, Cd63, Gata2, Hmmr, Pimreg, Cdc20, Tuba8 
       Ckap2l, Pbx1, Prc1, Cdca8, Ccnb2, Pdcd4, Rab27b, Tpx2, Malat1, Rgs18 
Negative:  Aif1, Ctss, Irf8, Cd74, Lsp1, Plbd1, Lgals3, H2-Aa, Cd52, Id2 
       Ifi205, H2-Eb1, Ccr2, Pld4, Batf3, Psap, Ighm, Mpeg1, H2-Ab1, Ckb 
       Ly86, Itgb7, Ms4a4c, Ifi30, Ctsh, Fth1, Naaa, Ms4a6c, Tmsb10, Jaml 
PC_ 5 
Positive:  Ftl1, S100a8, Pglyrp1, Clec4a2, Rgcc, Gstm1, S100a6, Trem3, Gda, Wfdc21 
       Lgals3, Dstn, Slfn2, Prdx5, Hp, H2-Eb1, Ly6c2, H2-Aa, Cd74, Cebpe 
       Gng11, Cd63, Mcemp1, Fcgr3, Ms4a3, Mmrn1, Mt1, Hacd4, Pdzk1ip1, Selenom 
Negative:  Stmn1, H2afy, Ptma, Plac8, Hsp90ab1, Tmsb10, Rps2, Cd34, Bcl2, Hspa8 
       Sox4, Npm1, Hist1h2ap, Cpa3, Cd48, Tespa1, Satb1, Ccl9, Fos, Ppia 
       Hist1h2ae, Hist1h2ac, Hmgb1, Lat2, Fabp5, BC035044, Jun, Dntt, Adgrg3, Egr1 

Plot results

VizDimLoadings(seurat.object, dims = 1:6, nfeatures = 10, reduction = "pca", ncol = 2)

DimPlot colored by orig.ident

DimPlot(seurat.object, reduction = "pca", group.by = "orig.ident")

Let’s put in a concerted effort to pick the right dimensionality using the newest software

# jackstraw.dim <- 40
# seurat.object <- JackStraw(seurat.object, num.replicate = 100, dims = jackstraw.dim) #runs ~50 min
# seurat.object <- ScoreJackStraw(seurat.object, dims = 1:jackstraw.dim)
# save.image(paste0(projectName, ".RData"))

Draw dim.reduction plots

# JackStrawPlot(seurat.object, dims = 25:36)
ElbowPlot(seurat.object, ndims = 50)
percent.variance(seurat.object@reductions$pca@stdev)

Number of PCs describing X% of variance

ElbowPlot(seurat.object, ndims = 50)

percent.variance(seurat.object@reductions$pca@stdev)

Add cluster IDs from Seurat v1

Exported cell IDs for clusters 3, 17, 10, 11 from Seurat v1. Will add these IDs as a metadata column.
Create column “clust.ID” and populate with 0’s. Then import IDs for clusters

tot.var <- percent.variance(seurat.object@reductions$pca@stdev, plot.var = FALSE, return.val = TRUE)
paste0("Num pcs for 80% variance:", length(which(cumsum(tot.var) <= 80)))
[1] "Num pcs for 80% variance:12"
paste0("Num pcs for 85% variance:", length(which(cumsum(tot.var) <= 85)))
[1] "Num pcs for 85% variance:18"
paste0("Num pcs for 90% variance:", length(which(cumsum(tot.var) <= 90)))
[1] "Num pcs for 90% variance:26"
paste0("Num pcs for 95% variance:", length(which(cumsum(tot.var) <= 95)))
[1] "Num pcs for 95% variance:37"

Add new metadata column and map new ids

clust3.cells <- read.table(file = "Seuratv1_clusterCellIDs/cluster3cellIDs.txt", col.names = "clust03")
clust3.cells <- sapply(clust3.cells, function(x) paste0(gsub("CMP", "CMPm2", x), "-1"))
clust17.cells <- read.table(file = "Seuratv1_clusterCellIDs/cluster17cellIDs.txt", col.names = "clust17")
clust17.cells <- sapply(clust17.cells, function(x) paste0(gsub("CMP", "CMPm2", x), "-1"))
clust10.cells <- read.table(file = "Seuratv1_clusterCellIDs/cluster10cellIDs.txt", col.names = "clust10")
clust10.cells <- sapply(clust10.cells, function(x) paste0(gsub("CMP", "CMPm2", x), "-1"))
clust11.cells <- read.table(file = "Seuratv1_clusterCellIDs/cluster11cellIDs.txt", col.names = "clust11")
clust11.cells <- sapply(clust11.cells, function(x) paste0(gsub("CMP", "CMPm2", x), "-1"))

do numbers make sense (we don’t expect the count to b exactly the same as the numbers in the original cluster)?

seurat.object@meta.data['clust.ID'] <- 0
seurat.object@meta.data$clust.ID[rownames(seurat.object@meta.data) %in% clust3.cells] <- 3
seurat.object@meta.data$clust.ID[rownames(seurat.object@meta.data) %in% clust17.cells] <- 17
seurat.object@meta.data$clust.ID[rownames(seurat.object@meta.data) %in% clust10.cells] <- 10
seurat.object@meta.data$clust.ID[rownames(seurat.object@meta.data) %in% clust11.cells] <- 11

make enough sense!

DGE

Let’s do some cluster analyses and see if we can find these populations in our new analysis. ## Ideal resolution…? (Is this a thing?) ### Color palette

nrow(seurat.object@meta.data[seurat.object@meta.data$clust.ID == 10,])
[1] 1049
nrow(seurat.object@meta.data[seurat.object@meta.data$clust.ID == 11,])
[1] 1118
nrow(seurat.object@meta.data[seurat.object@meta.data$clust.ID == 17,])
[1] 883
nrow(seurat.object@meta.data[seurat.object@meta.data$clust.ID == 3,])
[1] 1931

Total var 90%

Neighborhood and umap

set total.var <- 90%

color.palette <- c(
    "coral",
    "chartreuse4",
    "goldenrod1",
    "cadetblue1",
    "burlywood",
    "brown",
    "brown1",
    "blue",
    "blue4",
    "azure3",
    "aquamarine",
    "antiquewhite",
    "cadetblue",
    "gold3",
    "black",
    "darkgreen",
    "deeppink",
    "darkviolet",
    "darkturquoise",
    "darkslategray",
    "darksalmon",
    "darkorchid1",
    "darkolivegreen2",
    "forestgreen",
    "dodgerblue",
    "green",
    "lightpink",
    "lightcoral",
    "khaki1",
    "maroon",
    "peru",
    "lightseagreen",
    "lightsalmon",
    "plum",
    "moccasin",
    "tan",
    "tan1", 
    "red", 
    "purple",
    "khaki4",
    "black", 
    "plum4"
)

Plot UMAP

tot.var <- percent.variance(seurat.object@reductions$pca@stdev, plot.var = FALSE, return.val = TRUE)
ndims <- length(which(cumsum(tot.var) <= 90))
print(ndims)
[1] 26
seurat.object <- FindNeighbors(seurat.object, dims = 1:ndims)
Computing nearest neighbor graph
Computing SNN
seurat.object <- FindClusters(seurat.object, resolution = 0.5)
Modularity Optimizer version 1.3.0 by Ludo Waltman and Nees Jan van Eck

Number of nodes: 37104
Number of edges: 1255141

Running Louvain algorithm...
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Maximum modularity in 10 random starts: 0.8962
Number of communities: 13
Elapsed time: 12 seconds
seurat.object <- RunUMAP(seurat.object, dims = 1: ndims)
19:12:03 UMAP embedding parameters a = 0.9922 b = 1.112
19:12:03 Read 37104 rows and found 26 numeric columns
19:12:03 Using Annoy for neighbor search, n_neighbors = 30
19:12:03 Building Annoy index with metric = cosine, n_trees = 50
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
19:12:08 Writing NN index file to temp file /var/folders/4f/fwrj6fnn1dn4g8wsf0zv563hjsvl24/T//RtmptQx8Ib/filea01940da1192
19:12:08 Searching Annoy index using 1 thread, search_k = 3000
19:12:23 Annoy recall = 100%
19:12:24 Commencing smooth kNN distance calibration using 1 thread
19:12:26 Initializing from normalized Laplacian + noise
19:12:28 Commencing optimization for 200 epochs, with 1544862 positive edges
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
19:12:55 Optimization finished
# saveRDS(seurat.object, file = paste0(projectName, "_dim", ndims, ".RDS"))
for(x in c(0.5, 1, 1.5, 2, 2.5)){
    seurat.object <- FindClusters(seurat.object, resolution = x)
}
Modularity Optimizer version 1.3.0 by Ludo Waltman and Nees Jan van Eck

Number of nodes: 37104
Number of edges: 1255141

Running Louvain algorithm...
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Maximum modularity in 10 random starts: 0.8962
Number of communities: 13
Elapsed time: 12 seconds
Modularity Optimizer version 1.3.0 by Ludo Waltman and Nees Jan van Eck

Number of nodes: 37104
Number of edges: 1255141

Running Louvain algorithm...
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Maximum modularity in 10 random starts: 0.8602
Number of communities: 23
Elapsed time: 9 seconds
Modularity Optimizer version 1.3.0 by Ludo Waltman and Nees Jan van Eck

Number of nodes: 37104
Number of edges: 1255141

Running Louvain algorithm...
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Maximum modularity in 10 random starts: 0.8355
Number of communities: 30
Elapsed time: 9 seconds
Modularity Optimizer version 1.3.0 by Ludo Waltman and Nees Jan van Eck

Number of nodes: 37104
Number of edges: 1255141

Running Louvain algorithm...
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Maximum modularity in 10 random starts: 0.8156
Number of communities: 36
Elapsed time: 10 seconds
Modularity Optimizer version 1.3.0 by Ludo Waltman and Nees Jan van Eck

Number of nodes: 37104
Number of edges: 1255141

Running Louvain algorithm...
0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Maximum modularity in 10 random starts: 0.8006
Number of communities: 42
Elapsed time: 10 seconds

Generate statistics for each cluster/resolution combo

for (meta.col in colnames(seurat.object@meta.data)){
    if(grepl(pattern = ("RNA_snn_res"), x = meta.col)==TRUE){
        myplot <- DimPlot(seurat.object, 
                                            group.by = meta.col,
                                            reduction = "umap", 
                                            cols = color.palette
                                            ) + 
            ggtitle(paste0(projectName, " dim", ndims, "res", gsub("RNA_snn_res", "", meta.col) ))
        plot(myplot)
    }
}

For each resolution, what percentage of cells in each cluster are enriched for one of our clust.IDs?

Test: what percentage of each new clusterID matches one of the older clusters?

current_res <- 'RNA_snn_res.0.5'

cluster_ids <- sort(unique(seurat.object@meta.data[,current_res]))
counts_df <- data.frame(matrix(nrow = length(cluster_ids), ncol = 4))
rownames(counts_df) <- cluster_ids
colnames(counts_df) <- c("LSKm2", "CMPm2", "MEPm", "GMPm")
for(id in cluster_ids){
    cell_value <- nrow(seurat.object@meta.data[(seurat.object@meta.data[current_res] == id) & 
                                                                                            (seurat.object@meta.data$orig.ident == "LSKm2"),])
    counts_df[id, "LSKm2"] = cell_value
    cell_value <- nrow(seurat.object@meta.data[(seurat.object@meta.data[current_res] == id) & 
                                                                                            (seurat.object@meta.data$orig.ident == "LSKm2"),])
    counts_df[id, "CMPm2"] = cell_value
    cell_value <- nrow(seurat.object@meta.data[(seurat.object@meta.data[current_res] == id) & 
                                                                                            (seurat.object@meta.data$orig.ident == "MEPm"),])
    counts_df[id, "MEPm"] = cell_value
    cell_value <- nrow(seurat.object@meta.data[(seurat.object@meta.data[current_res] == id) & 
                                                                                            (seurat.object@meta.data$orig.ident == "GMPm"),])
    counts_df[id, "GMPm"] = cell_value

}

Absolutely terrible overlap, no enrichment of any of these across the new clustering algorithm. Maybe should try 95% variation covered

Find old cells on UMAP

time for the super scarey moment to see if the cells from seuratv1 still cluster together on in seurat v4

for (meta.col in colnames(seurat.object@meta.data)){
    if(grepl(pattern = ("RNA_snn_res"), x = meta.col)==TRUE){
        new.clusters <- sort(as.numeric(levels(seurat.object@meta.data[[meta.col]])))
        enrich.df <- data.frame(matrix(ncol = 4, nrow = length(new.clusters)))
        colnames(enrich.df) <- c(3, 17, 10, 11)
        rownames(enrich.df) <- new.clusters
        meta.df <- seurat.object@meta.data
        for(row.id in rownames(enrich.df)){
            tot.clus <- nrow(meta.df[meta.df[[meta.col]] == row.id,])
            for(col.id in colnames(enrich.df)){
                num.x <- nrow(meta.df[(meta.df[[meta.col]] == row.id) & (meta.df$clust.ID == col.id),])
                pct.x <- as.integer(num.x / tot.clus *100)
                # print(pct.x)
                enrich.df[row.id, col.id] <- pct.x
            }
        }
        colnames(enrich.df) <- sapply(colnames(enrich.df), function(x) paste0("oldcluster", x))
        rownames(enrich.df) <- sapply(rownames(enrich.df), function(x) paste0("newcluster", x))
        xlsx::write.xlsx(enrich.df, file = paste0("PctOfNewClustersOverlappingOldClusters_", projectName, "_dim", ndims, ".xlsx"), sheetName = paste0(gsub("RNA_snn_", "", meta.col)), append = TRUE)
        print(enrich.df)
    }
}
Error in .jcall(wb, "Lorg/apache/poi/ss/usermodel/Sheet;", "createSheet",  : 
  java.lang.IllegalArgumentException: The workbook already contains a sheet of this name
DimPlot(seurat.object,
                reduction = "umap",
                group.by = "clust.ID", 
                # split.by = "orig.ident",
                cols = c("gray", "orange", "blue", "red", "green"),)

Total var 95%

Neighborhood and umap

set total.var <- 95%

DimPlot(seurat.object,
                reduction = "umap",
                group.by = "orig.ident", 
                split.by = "clust.ID",
                cols = c("gray", "orange", "blue", "red", "green"),)

Plot UMAP

tot.var <- percent.variance(seurat.object@reductions$pca@stdev, plot.var = FALSE, return.val = TRUE)
ndims <- length(which(cumsum(tot.var) <= 95))
print(ndims)
[1] 37
seurat.object <- FindNeighbors(seurat.object, dims = 1:ndims)
Computing nearest neighbor graph

For each resolution, what percentage of cells in each cluster are enriched for one of our clust.IDs?

Test: what percentage of each new clusterID matches one of the older clusters?

Absolutely terrible overlap, no enrichment of any of these across the new clustering algorithm. Maybe should try 95% variation covered

Find old cells on UMAP

time for the super scarey moment to see if the cells from seuratv1 still cluster together on in seurat v4

Gene expression of old clustrs on new map

Let’s see if we can get some gene expression profiles on these…

Evaluate cluster stability

Must ensure we have the right cluster stability, that is, cells that start in the same cluster tend to stay in the same cluster. If your data is over-clustered, cells will bounce between groups.

Following [this tutorial by Matt O.].https://towardsdatascience.com/10-tips-for-choosing-the-optimal-number-of-clusters-277e93d72d92.

Clustree

Previously my favourite has been Clustree, which gives a nice visual NB: For some reason clustree::clustree() didn’t work, whereas library(clustree) followed by clustree() did.

These data suggest that node stability is aweful! Need to figure out if this is a dimensional reduction error or a clustering error.

Why are clusters so unstable?

Differences could include: * cells in each population (cellranger v6 includes more cells than cellranger v1, especially in MEP) * dimensionality is incorrect * ScaleData didnt account for regression factors (e.g., “nCounts_RNA” or “nFeatures_RNA”) * Did not consider cell cycle * Incorrect normalization/scaling method * Clustering is too strict or not strict enough * neighborhood analysis used wrong parameters * Should include mitoC filter (there’s a chunk of MEP w/ mitoC @ ~40%) * SCTransform accounts better for sources of variability

Looks like MEPm is the only sample with that huge MitoC % lump @ 40%. What do these cells look like, otherwise?

Save dim36 as is and try clustering analysis @ dim24

Repeat clustering with dim24

One possibility is that I’ve included too many dimensions. Will see if 90% increases stability.

Save object

Evaluate cluster stability

Must ensure we have the right cluster stability, that is, cells that start in the same cluster tend to stay in the same cluster. If your data is over-clustered, cells will bounce between groups.

Following [this tutorial by Matt O.].https://towardsdatascience.com/10-tips-for-choosing-the-optimal-number-of-clusters-277e93d72d92.

Clustree

Previously my favourite has been Clustree, which gives a nice visual NB: For some reason clustree::clustree() didn’t work, whereas library(clustree) followed by clustree() did.

Think I’ll explore regression factors using SCTransform in new document.

LS0tCnRpdGxlOiAibXNBZ2dyX3NldXJhdCIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKQ3JlYXRpbmcgbmV3IHBpcGVsaW5lIHVzaW5nIHNldXJhdCB2NC4wLjIgYXZhaWxhYmxlIDIwMjEuMDYuMjMgIApJbXBvcnRhbnQgbm90ZXM6CgoqIEZJTFRFUklORyBvbiBgcGVyY2VudC5tdGAsIGJ1dCBOT1QgcmVncmVzc2luZyBvbiBgcGVyY2VudC5tdGAKKiBSZWdyZXNzaW5nIG9uIGBuQ291bnRzX1JOQWAgYW5kIGBuRmVhdHVyZV9STkFgCgpMb2FkIGxpYnJhcmllcyByZXF1aXJlZCBmb3IgU2V1cmF0djQKCmBgYHtyIHNldHVwLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQprbml0cjo6b3B0c19rbml0JHNldChyb290LmRpciA9ICJ+L0Rlc2t0b3AvMTBYR2Vub21pY3NEYXRhL21zQWdncl9zY1JOQVNlcS8iKQpsaWJyYXJ5KGRwbHlyKQpsaWJyYXJ5KFNldXJhdCkKbGlicmFyeShwYXRjaHdvcmspCmxpYnJhcnkoZ2dwbG90MikKIyBsaWJyYXJ5KGNsdXN0cmVlKQpgYGAKc3RvcmUgc2Vzc2lvbiBpbmZvCmBgYHtyfQpzaW5rKCJtc0FnZ3Jfc2V1cmF0LXYxLjIwMjEwNjIzIikKc2Vzc2lvbkluZm8oKQpzaW5rKCkKYGBgCgojIEEgbm90ZSBhYm91dCB1c2luZyBTQ1RyYW5zZm9ybSB2ZXJzdXMgYFNjYWxlRGF0YWAKaHR0cHM6Ly9iaW9jb25kdWN0b3Iub3JnL3BhY2thZ2VzLzMuMTAvd29ya2Zsb3dzL3ZpZ25ldHRlcy9zaW1wbGVTaW5nbGVDZWxsL2luc3QvZG9jL2JhdGNoLmh0bWwjNjJfZm9yX2dlbmUtYmFzZWRfYW5hbHlzZXMKPllvdSBjYW4gYWxzbyBub3JtYWxpemUgYW5kIHNjYWxlIGRhdGEgZm9yIHRoZSBSTkEgYXNzYXkuIFRoZXJlIGFyZSBudW1lcm91cyByZXNvdXJjZXMgb24gdGhpcywgYnV0IEFhcm9uIEx1biBkZXNjcmliZXMgd2h5IHRoZSBvcmlnaW5hbCBsb2ctbm9ybWFsaXplZCB2YWx1ZXMgc2hvdWxkIGJlIHVzZWQgZm9yIERFIGFuZCB2aXN1YWxpemF0aW9ucyBvZiBleHByZXNzaW9uIHF1aXRlIHdlbGwgaGVyZToKPgo+Rm9yIGdlbmUtYmFzZWQgcHJvY2VkdXJlcyBsaWtlIGRpZmZlcmVudGlhbCBleHByZXNzaW9uIChERSkgYW5hbHlzZXMgb3IgZ2VuZSBuZXR3b3JrIGNvbnN0cnVjdGlvbiwgaXQgaXMgZGVzaXJhYmxlIHRvIHVzZSB0aGUgb3JpZ2luYWwgbG9nLWV4cHJlc3Npb24gdmFsdWVzIG9yIGNvdW50cy4gVGhlIGNvcnJlY3RlZCB2YWx1ZXMgYXJlIG9ubHkgdXNlZCB0byBvYnRhaW4gY2VsbC1sZXZlbCByZXN1bHRzIHN1Y2ggYXMgY2x1c3RlcnMgb3IgdHJhamVjdG9yaWVzLiBCYXRjaCBlZmZlY3RzIGFyZSBoYW5kbGVkIGV4cGxpY2l0bHkgdXNpbmcgYmxvY2tpbmcgdGVybXMgb3IgdmlhIGEgbWV0YS1hbmFseXNpcyBhY3Jvc3MgYmF0Y2hlcy4gV2UgZG8gbm90IHVzZSB0aGUgY29ycmVjdGVkIHZhbHVlcyBkaXJlY3RseSBpbiBnZW5lLWJhc2VkIGFuYWx5c2VzLCBmb3IgdmFyaW91cyByZWFzb25zOgo+Cj5JdCBpcyB1c3VhbGx5IGluYXBwcm9wcmlhdGUgdG8gcGVyZm9ybSBERSBhbmFseXNlcyBvbiBiYXRjaC1jb3JyZWN0ZWQgdmFsdWVzLCBkdWUgdG8gdGhlIGZhaWx1cmUgdG8gbW9kZWwgdGhlIHVuY2VydGFpbnR5IG9mIHRoZSBjb3JyZWN0aW9uLiBUaGlzIHVzdWFsbHkgcmVzdWx0cyBpbiBsb3NzIG9mIHR5cGUgSSBlcnJvciBjb250cm9sLCBpLmUuLCBtb3JlIGZhbHNlIHBvc2l0aXZlcyB0aGFuIGV4cGVjdGVkLgo+Cj5UaGUgY29ycmVjdGlvbiBkb2VzIG5vdCBwcmVzZXJ2ZSB0aGUgbWVhbi12YXJpYW5jZSByZWxhdGlvbnNoaXAuIEFwcGxpY2F0aW9ucyBvZiBjb21tb24gREUgbWV0aG9kcyBsaWtlIGVkZ2VSIG9yIGxpbW1hIGFyZSB1bmxpa2VseSB0byBiZSB2YWxpZC4KPgo+QmF0Y2ggY29ycmVjdGlvbiBtYXkgKGNvcnJlY3RseSkgcmVtb3ZlIGJpb2xvZ2ljYWwgZGlmZmVyZW5jZXMgYmV0d2VlbiBiYXRjaGVzIGluIHRoZSBjb3Vyc2Ugb2YgbWFwcGluZyBhbGwgY2VsbHMgb250byBhIGNvbW1vbiBjb29yZGluYXRlIHN5c3RlbS4gUmV0dXJuaW5nIHRvIHRoZSB1bmNvcnJlY3RlZCBleHByZXNzaW9uIHZhbHVlcyBwcm92aWRlcyBhbiBvcHBvcnR1bml0eSBmb3IgZGV0ZWN0aW5nIHN1Y2ggZGlmZmVyZW5jZXMgaWYgdGhleSBhcmUgb2YgaW50ZXJlc3QuIENvbnZlcnNlbHksIGlmIHRoZSBiYXRjaCBjb3JyZWN0aW9uIG1hZGUgYSBtaXN0YWtlLCB0aGUgdXNlIG9mIHRoZSB1bmNvcnJlY3RlZCBleHByZXNzaW9uIHZhbHVlcyBwcm92aWRlcyBhbiBpbXBvcnRhbnQgc2FuaXR5IGNoZWNrLgo+Cj5JbiBhZGRpdGlvbiwgdGhlIG5vcm1hbGl6ZWQgdmFsdWVzIGluIFNDVCBhbmQgaW50ZWdyYXRlZCBhc3NheXMgZG9uJ3QgbmVjZXNzYXJ5IGNvcnJlc3BvbmQgdG8gcGVyLWdlbmUgZXhwcmVzc2lvbiB2YWx1ZXMgYW55d2F5LCByYXRoZXIgY29udGFpbmluZyByZXNpZHVhbHMgKGluIHRoZSBjYXNlIG9mIHRoZSBzY2FsZS5kYXRhIHNsb3QgZm9yIGVhY2gpLgoKCgpNZXNzIHdpdGggaG93IHRvIGxvYWQgNCBjZWxsIHBvcHVsYXRpb25zIGludG8gc2luZ2xlIHNldXJhdCBvYmplY3QKCgoKU0VUIFNFRUQ/Pz8/PyEhISEhCgojIyBTZXQgZ2xvYmFsIHZhcmlhYmxlcwoKYGBge3J9CnByb2plY3ROYW1lIDwtICJtc0FnZ3IiCmphY2tzdHJhdy5kaW0gPC0gNDAKYGBgCgoKCmBgYHtyfQpzb3VyY2UoIm1zQWdncl9BbmFseXNpc0NvZGUvcmVhZF8xMFhHZW5vbWljc19kYXRhLlIiKQpzb3VyY2UoIm1zQWdncl9BbmFseXNpc0NvZGUvUGVyY2VudFZhcmlhbmNlLlIiKQpgYGAKCgpgYGB7cn0Kc2V0d2QoIi4uL2NlbGxSYW5nZXIvIikgIyB0ZW1wb3JhcmlseSBjaGFuZ2luZyB3ZCBvbmx5IHdvcmtzIGlmIHlvdSBydW4gdGhlIGVudGlyZSBjaHVuayBhdCBvbmNlCmRhdGFfZmlsZS5saXN0IDwtIHJlYWRfMTBYR2Vub21pY3NfZGF0YShzYW1wbGUubGlzdCA9IGMoIkxTS20yIiwgIkNNUG0yIiwgIk1FUG0iLCAiR01QbSIpKQpkYXRhLm9iamVjdDwtUmVhZDEwWChkYXRhX2ZpbGUubGlzdCkKYGBgCgoKCmBgYHtyfQpzZXVyYXQub2JqZWN0PC0gQ3JlYXRlU2V1cmF0T2JqZWN0KGNvdW50cyA9IGRhdGEub2JqZWN0LCBtaW4uY2VsbHMgPSAzLCBtaW4uZ2VuZXMgPSAyMDAsIHByb2plY3QgPSBwcm9qZWN0TmFtZSkKYGBgCgpDbGVhbiB1cCB0byBmcmVlIG1lbW9yeQoKYGBge3J9CnJlbW92ZShkYXRhLm9iamVjdCkKYGBgCgoKQWRkIG1pdG9jaG9uZHJpYWwgbWV0YWRhdGEgYW5kIHBsb3Qgc29tZSBiYXNpYyBmZWF0dXJlcwpgYGB7cn0Kc2V1cmF0Lm9iamVjdFtbInBlcmNlbnQubXQiXV0gPC0gUGVyY2VudGFnZUZlYXR1cmVTZXQoc2V1cmF0Lm9iamVjdCwgcGF0dGVybiA9ICJebXQtIikKVmxuUGxvdChzZXVyYXQub2JqZWN0LCBmZWF0dXJlcyA9IGMoIm5GZWF0dXJlX1JOQSIsICJuQ291bnRfUk5BIiwgInBlcmNlbnQubXQiKSwgbmNvbCA9IDMsIHB0LnNpemUgPSAwLCBmaWxsLmJ5ID0gJ29yaWcuaWRlbnQnLCApCmBgYAoKCmBgYHtyIGZpZy53aWR0aD00LCBmaWcuaGVpZ2h0PTJ9CnBsb3QxIDwtIEZlYXR1cmVTY2F0dGVyKHNldXJhdC5vYmplY3QsIGZlYXR1cmUxID0gIm5Db3VudF9STkEiLCBmZWF0dXJlMiA9ICJwZXJjZW50Lm10IiwgZ3JvdXAuYnkgPSAib3JpZy5pZGVudCIsIHB0LnNpemUgPSAwLjAxKQpwbG90MiA8LSBGZWF0dXJlU2NhdHRlcihzZXVyYXQub2JqZWN0LCBmZWF0dXJlMSA9ICJuQ291bnRfUk5BIiwgZmVhdHVyZTIgPSAibkZlYXR1cmVfUk5BIiwgZ3JvdXAuYnkgPSAib3JpZy5pZGVudCIsIHB0LnNpemUgPSAwLjAxKQpwbG90MSArIHBsb3QyCmBgYAoKCnJlbW92ZSBsb3cgcXVhbGl0eSBjZWxscwpyZXF1aXJlOiBuRmVhdHVyZV9STkEgYmV0d2VlbiAyMDAgYW5kIDQwMDAgKGluY2x1c2l2ZSkKcmVxdWlyZTogcGVyY2VudC5tdCA8PSA1CgpgYGB7cn0KcHJpbnQocGFzdGUoIm9yaWdpbmFsIG9iamVjdDoiLCBucm93KHNldXJhdC5vYmplY3RAbWV0YS5kYXRhKSwgImNlbGxzIiwgc2VwID0gIiAiKSkKc2V1cmF0Lm9iamVjdCA8LSBzdWJzZXQoc2V1cmF0Lm9iamVjdCwgCgkJCQkJCQkJCQkJCXN1YnNldCA9IG5GZWF0dXJlX1JOQSA+PTIwMCAmIAoJCQkJCQkJCQkJCQkJbkZlYXR1cmVfUk5BIDw9IDQwMDAgJiAKCQkJCQkJCQkJCQkJCXBlcmNlbnQubXQgPD0gNQoJCQkJCQkJCQkJCQkpCnByaW50KHBhc3RlKCJuZXcgb2JqZWN0OiIsIG5yb3coc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGEpLCAiY2VsbHMiLCBzZXAgPSAiICIpKQpgYGAKCgoKIyMgTm9ybWFsaXphdGlvbgoKU3RydWdnbGluZyB0byB3cmFwIG15IGhlYWQgYXJvdW5kIHRoaXMgb25lLiBJdCBzZWVtcyB0aGF0IFNDVHJhbnNmb3JtIGlzIGJlc3QgZm9yIGJhdGNoIGNvcnJlY3Rpb24sIGJ1dCBgTm9ybWFsaXplRGF0YWAgYW5kIGBTY2FsZURhdGFgIGFyZSBiZXN0IGZvciBER0UuIFNldmVyYWwgdmlnbmV0dGVzIGhhdmUgcGVyZm9ybWVkIGJvdGgKCmBzZWxlY3Rpb24ubWV0aG9kCQpIb3cgdG8gY2hvb3NlIHRvcCB2YXJpYWJsZSBmZWF0dXJlcy4gQ2hvb3NlIG9uZSBvZiA6Cgp2c3Q6IEZpcnN0LCBmaXRzIGEgbGluZSB0byB0aGUgcmVsYXRpb25zaGlwIG9mIGxvZyh2YXJpYW5jZSkgYW5kIGxvZyhtZWFuKSB1c2luZyBsb2NhbCBwb2x5bm9taWFsIHJlZ3Jlc3Npb24gKGxvZXNzKS4gVGhlbiBzdGFuZGFyZGl6ZXMgdGhlIGZlYXR1cmUgdmFsdWVzIHVzaW5nIHRoZSBvYnNlcnZlZCBtZWFuIGFuZCBleHBlY3RlZCB2YXJpYW5jZSAoZ2l2ZW4gYnkgdGhlIGZpdHRlZCBsaW5lKS4gRmVhdHVyZSB2YXJpYW5jZSBpcyB0aGVuIGNhbGN1bGF0ZWQgb24gdGhlIHN0YW5kYXJkaXplZCB2YWx1ZXMgYWZ0ZXIgY2xpcHBpbmcgdG8gYSBtYXhpbXVtIChzZWUgY2xpcC5tYXggcGFyYW1ldGVyKS4KCm1lYW4udmFyLnBsb3QgKG12cCk6IEZpcnN0LCB1c2VzIGEgZnVuY3Rpb24gdG8gY2FsY3VsYXRlIGF2ZXJhZ2UgZXhwcmVzc2lvbiAobWVhbi5mdW5jdGlvbikgYW5kIGRpc3BlcnNpb24gKGRpc3BlcnNpb24uZnVuY3Rpb24pIGZvciBlYWNoIGZlYXR1cmUuIE5leHQsIGRpdmlkZXMgZmVhdHVyZXMgaW50byBudW0uYmluIChkZWFmdWx0IDIwKSBiaW5zIGJhc2VkIG9uIHRoZWlyIGF2ZXJhZ2UgZXhwcmVzc2lvbiwgYW5kIGNhbGN1bGF0ZXMgei1zY29yZXMgZm9yIGRpc3BlcnNpb24gd2l0aGluIGVhY2ggYmluLiBUaGUgcHVycG9zZSBvZiB0aGlzIGlzIHRvIGlkZW50aWZ5IHZhcmlhYmxlIGZlYXR1cmVzIHdoaWxlIGNvbnRyb2xsaW5nIGZvciB0aGUgc3Ryb25nIHJlbGF0aW9uc2hpcCBiZXR3ZWVuIHZhcmlhYmlsaXR5IGFuZCBhdmVyYWdlIGV4cHJlc3Npb24uCgpkaXNwZXJzaW9uIChkaXNwKTogc2VsZWN0cyB0aGUgZ2VuZXMgd2l0aCB0aGUgaGlnaGVzdCBkaXNwZXJzaW9uIHZhbHVlc2AKCgoKCmBgYHtyfQpzZXVyYXQub2JqZWN0IDwtIE5vcm1hbGl6ZURhdGEoc2V1cmF0Lm9iamVjdCwgbm9ybWFsaXphdGlvbi5tZXRob2QgPSAiTG9nTm9ybWFsaXplIiwgc2NhbGUuZmFjdG9yID0gMTAwMDApCmBgYAoKCgoKCkZpbmQgdmFyaWFibGUgZmVhdHVyZXMKYGBge3IgZmlnLndpZHRoID0gNSwgZmlnLmhlaWdodCA9IDJ9CnNldXJhdC5vYmplY3QgPC0gRmluZFZhcmlhYmxlRmVhdHVyZXMoc2V1cmF0Lm9iamVjdCwgc2VsZWN0aW9uLm1ldGhvZCA9ICJ2c3QiLCBuZmVhdHVyZXMgPSAyMDAwKQp0b3AxMCA8LSBoZWFkKFZhcmlhYmxlRmVhdHVyZXMoc2V1cmF0Lm9iamVjdCksIDEwKQpwbG90MSA8LSBWYXJpYWJsZUZlYXR1cmVQbG90KHNldXJhdC5vYmplY3QpCnBsb3QyIDwtIExhYmVsUG9pbnRzKHBsb3QgPSBwbG90MSwgcG9pbnRzID0gdG9wMTAsIHJlcGVsID0gVFJVRSkKcGxvdDEgKyBwbG90MgpgYGAKClNjYWxlIGRhdGEgKGxpbmVhciB0cmFuc2Zvcm1hdGlvbikKCmBgYHtyfQphbGwuZ2VuZXMgPC0gcm93bmFtZXMoc2V1cmF0Lm9iamVjdCkKc2V1cmF0Lm9iamVjdCA8LSBTY2FsZURhdGEoc2V1cmF0Lm9iamVjdCwgZmVhdHVyZXMgPSBhbGwuZ2VuZXMsIHZhcnMudG8ucmVncmVzcyA9IGMoIm5Db3VudF9STkEiLCAibkZlYXR1cmVfUk5BIikpCmBgYAoKCiMjIyBTYXZlIHByb2dyZXNzCgpgYGB7cn0Kc2F2ZS5pbWFnZShmaWxlID0gcGFzdGUwKHByb2plY3ROYW1lLCAnLlJEYXRhJykpCmBgYAoKCiMjIFBDQQoKbGluZWFyIGRpbWVuc2lvbmFsIHJlZHVjdGlvbi4gRGVmYXVsdCBhcmUgYmFzZWQgb24gVmFyaWFibGVGZWF0dXJlcywgYnV0IGNhbiBiZSBjaGFuZ2VkCgpgYGB7cn0Kc2V1cmF0Lm9iamVjdCA8LSBSdW5QQ0Eoc2V1cmF0Lm9iamVjdCwgZmVhdHVyZXMgPSBWYXJpYWJsZUZlYXR1cmVzKG9iamVjdCA9IHNldXJhdC5vYmplY3QpKQpgYGAKUGxvdCByZXN1bHRzCmBgYHtyIGZpZy53aWR0aD00LCBmaWcuaGVpZ2h0PTR9ClZpekRpbUxvYWRpbmdzKHNldXJhdC5vYmplY3QsIGRpbXMgPSAxOjYsIG5mZWF0dXJlcyA9IDEwLCByZWR1Y3Rpb24gPSAicGNhIiwgbmNvbCA9IDIpCmBgYAoKRGltUGxvdCBjb2xvcmVkIGJ5IG9yaWcuaWRlbnQKYGBge3J9CkRpbVBsb3Qoc2V1cmF0Lm9iamVjdCwgcmVkdWN0aW9uID0gInBjYSIsIGdyb3VwLmJ5ID0gIm9yaWcuaWRlbnQiKQpgYGAKTGV0J3MgcHV0IGluIGEgY29uY2VydGVkIGVmZm9ydCB0byBwaWNrIHRoZSByaWdodCBkaW1lbnNpb25hbGl0eSB1c2luZyB0aGUgbmV3ZXN0IHNvZnR3YXJlCmBgYHtyfQojIGphY2tzdHJhdy5kaW0gPC0gNDAKIyBzZXVyYXQub2JqZWN0IDwtIEphY2tTdHJhdyhzZXVyYXQub2JqZWN0LCBudW0ucmVwbGljYXRlID0gMTAwLCBkaW1zID0gamFja3N0cmF3LmRpbSkgI3J1bnMgfjUwIG1pbgojIHNldXJhdC5vYmplY3QgPC0gU2NvcmVKYWNrU3RyYXcoc2V1cmF0Lm9iamVjdCwgZGltcyA9IDE6amFja3N0cmF3LmRpbSkKIyBzYXZlLmltYWdlKHBhc3RlMChwcm9qZWN0TmFtZSwgIi5SRGF0YSIpKQpgYGAKRHJhdyBkaW0ucmVkdWN0aW9uIHBsb3RzCmBgYHtyfQojIEphY2tTdHJhd1Bsb3Qoc2V1cmF0Lm9iamVjdCwgZGltcyA9IDI1OjM2KQpgYGAKYGBge3IsIGZpZ3VyZXMtc2lkZSwgZmlnLnNob3c9J2hvbGQnLCBvdXQud2lkdGg9IjUwJSJ9CkVsYm93UGxvdChzZXVyYXQub2JqZWN0LCBuZGltcyA9IDUwKQpwZXJjZW50LnZhcmlhbmNlKHNldXJhdC5vYmplY3RAcmVkdWN0aW9ucyRwY2FAc3RkZXYpCmBgYApOdW1iZXIgb2YgUENzIGRlc2NyaWJpbmcgWCUgb2YgdmFyaWFuY2UKYGBge3J9CnRvdC52YXIgPC0gcGVyY2VudC52YXJpYW5jZShzZXVyYXQub2JqZWN0QHJlZHVjdGlvbnMkcGNhQHN0ZGV2LCBwbG90LnZhciA9IEZBTFNFLCByZXR1cm4udmFsID0gVFJVRSkKcGFzdGUwKCJOdW0gcGNzIGZvciA4MCUgdmFyaWFuY2U6IiwgbGVuZ3RoKHdoaWNoKGN1bXN1bSh0b3QudmFyKSA8PSA4MCkpKQpwYXN0ZTAoIk51bSBwY3MgZm9yIDg1JSB2YXJpYW5jZToiLCBsZW5ndGgod2hpY2goY3Vtc3VtKHRvdC52YXIpIDw9IDg1KSkpCnBhc3RlMCgiTnVtIHBjcyBmb3IgOTAlIHZhcmlhbmNlOiIsIGxlbmd0aCh3aGljaChjdW1zdW0odG90LnZhcikgPD0gOTApKSkKcGFzdGUwKCJOdW0gcGNzIGZvciA5NSUgdmFyaWFuY2U6IiwgbGVuZ3RoKHdoaWNoKGN1bXN1bSh0b3QudmFyKSA8PSA5NSkpKQoKYGBgCgojIyBBZGQgY2x1c3RlciBJRHMgZnJvbSBTZXVyYXQgdjEKCkV4cG9ydGVkIGNlbGwgSURzIGZvciBjbHVzdGVycyAzLCAxNywgMTAsIDExIGZyb20gU2V1cmF0IHYxLiBXaWxsIGFkZCB0aGVzZSBJRHMgYXMgYSBtZXRhZGF0YSBjb2x1bW4uICAKQ3JlYXRlIGNvbHVtbiAiY2x1c3QuSUQiIGFuZCBwb3B1bGF0ZSB3aXRoIDAncy4gVGhlbiBpbXBvcnQgSURzIGZvciBjbHVzdGVycwpgYGB7cn0KY2x1c3QzLmNlbGxzIDwtIHJlYWQudGFibGUoZmlsZSA9ICJTZXVyYXR2MV9jbHVzdGVyQ2VsbElEcy9jbHVzdGVyM2NlbGxJRHMudHh0IiwgY29sLm5hbWVzID0gImNsdXN0MDMiKQpjbHVzdDMuY2VsbHMgPC0gc2FwcGx5KGNsdXN0My5jZWxscywgZnVuY3Rpb24oeCkgcGFzdGUwKGdzdWIoIkNNUCIsICJDTVBtMiIsIHgpLCAiLTEiKSkKY2x1c3QxNy5jZWxscyA8LSByZWFkLnRhYmxlKGZpbGUgPSAiU2V1cmF0djFfY2x1c3RlckNlbGxJRHMvY2x1c3RlcjE3Y2VsbElEcy50eHQiLCBjb2wubmFtZXMgPSAiY2x1c3QxNyIpCmNsdXN0MTcuY2VsbHMgPC0gc2FwcGx5KGNsdXN0MTcuY2VsbHMsIGZ1bmN0aW9uKHgpIHBhc3RlMChnc3ViKCJDTVAiLCAiQ01QbTIiLCB4KSwgIi0xIikpCmNsdXN0MTAuY2VsbHMgPC0gcmVhZC50YWJsZShmaWxlID0gIlNldXJhdHYxX2NsdXN0ZXJDZWxsSURzL2NsdXN0ZXIxMGNlbGxJRHMudHh0IiwgY29sLm5hbWVzID0gImNsdXN0MTAiKQpjbHVzdDEwLmNlbGxzIDwtIHNhcHBseShjbHVzdDEwLmNlbGxzLCBmdW5jdGlvbih4KSBwYXN0ZTAoZ3N1YigiQ01QIiwgIkNNUG0yIiwgeCksICItMSIpKQpjbHVzdDExLmNlbGxzIDwtIHJlYWQudGFibGUoZmlsZSA9ICJTZXVyYXR2MV9jbHVzdGVyQ2VsbElEcy9jbHVzdGVyMTFjZWxsSURzLnR4dCIsIGNvbC5uYW1lcyA9ICJjbHVzdDExIikKY2x1c3QxMS5jZWxscyA8LSBzYXBwbHkoY2x1c3QxMS5jZWxscywgZnVuY3Rpb24oeCkgcGFzdGUwKGdzdWIoIkNNUCIsICJDTVBtMiIsIHgpLCAiLTEiKSkKYGBgCgpBZGQgbmV3IG1ldGFkYXRhIGNvbHVtbiBhbmQgbWFwIG5ldyBpZHMKYGBge3J9CnNldXJhdC5vYmplY3RAbWV0YS5kYXRhWydjbHVzdC5JRCddIDwtIDAKc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGEkY2x1c3QuSURbcm93bmFtZXMoc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGEpICVpbiUgY2x1c3QzLmNlbGxzXSA8LSAzCnNldXJhdC5vYmplY3RAbWV0YS5kYXRhJGNsdXN0LklEW3Jvd25hbWVzKHNldXJhdC5vYmplY3RAbWV0YS5kYXRhKSAlaW4lIGNsdXN0MTcuY2VsbHNdIDwtIDE3CnNldXJhdC5vYmplY3RAbWV0YS5kYXRhJGNsdXN0LklEW3Jvd25hbWVzKHNldXJhdC5vYmplY3RAbWV0YS5kYXRhKSAlaW4lIGNsdXN0MTAuY2VsbHNdIDwtIDEwCnNldXJhdC5vYmplY3RAbWV0YS5kYXRhJGNsdXN0LklEW3Jvd25hbWVzKHNldXJhdC5vYmplY3RAbWV0YS5kYXRhKSAlaW4lIGNsdXN0MTEuY2VsbHNdIDwtIDExCmBgYAoKZG8gbnVtYmVycyBtYWtlIHNlbnNlICh3ZSBkb24ndCBleHBlY3QgdGhlIGNvdW50IHRvIGIgZXhhY3RseSB0aGUgc2FtZSBhcyB0aGUgbnVtYmVycyBpbiB0aGUgb3JpZ2luYWwgY2x1c3Rlcik/CmBgYHtyfQpucm93KHNldXJhdC5vYmplY3RAbWV0YS5kYXRhW3NldXJhdC5vYmplY3RAbWV0YS5kYXRhJGNsdXN0LklEID09IDEwLF0pCm5yb3coc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGFbc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGEkY2x1c3QuSUQgPT0gMTEsXSkKbnJvdyhzZXVyYXQub2JqZWN0QG1ldGEuZGF0YVtzZXVyYXQub2JqZWN0QG1ldGEuZGF0YSRjbHVzdC5JRCA9PSAxNyxdKQpucm93KHNldXJhdC5vYmplY3RAbWV0YS5kYXRhW3NldXJhdC5vYmplY3RAbWV0YS5kYXRhJGNsdXN0LklEID09IDMsXSkKYGBgCgptYWtlIGVub3VnaCBzZW5zZSEKCgoKIyBER0UKCkxldCdzIGRvIHNvbWUgY2x1c3RlciBhbmFseXNlcyBhbmQgc2VlIGlmIHdlIGNhbiBmaW5kIHRoZXNlIHBvcHVsYXRpb25zIGluIG91ciBuZXcgYW5hbHlzaXMuCiMjIElkZWFsIHJlc29sdXRpb24uLi4/IChJcyB0aGlzIGEgdGhpbmc/KQojIyMgQ29sb3IgcGFsZXR0ZQpgYGB7cn0KY29sb3IucGFsZXR0ZSA8LSBjKAoJImNvcmFsIiwKCSJjaGFydHJldXNlNCIsCgkiZ29sZGVucm9kMSIsCgkiY2FkZXRibHVlMSIsCgkiYnVybHl3b29kIiwKCSJicm93biIsCgkiYnJvd24xIiwKCSJibHVlIiwKCSJibHVlNCIsCgkiYXp1cmUzIiwKCSJhcXVhbWFyaW5lIiwKCSJhbnRpcXVld2hpdGUiLAoJImNhZGV0Ymx1ZSIsCgkiZ29sZDMiLAoJImJsYWNrIiwKCSJkYXJrZ3JlZW4iLAoJImRlZXBwaW5rIiwKCSJkYXJrdmlvbGV0IiwKCSJkYXJrdHVycXVvaXNlIiwKCSJkYXJrc2xhdGVncmF5IiwKCSJkYXJrc2FsbW9uIiwKCSJkYXJrb3JjaGlkMSIsCgkiZGFya29saXZlZ3JlZW4yIiwKCSJmb3Jlc3RncmVlbiIsCgkiZG9kZ2VyYmx1ZSIsCgkiZ3JlZW4iLAoJImxpZ2h0cGluayIsCgkibGlnaHRjb3JhbCIsCgkia2hha2kxIiwKCSJtYXJvb24iLAoJInBlcnUiLAoJImxpZ2h0c2VhZ3JlZW4iLAoJImxpZ2h0c2FsbW9uIiwKCSJwbHVtIiwKCSJtb2NjYXNpbiIsCgkidGFuIiwKCSJ0YW4xIiwgCgkicmVkIiwgCgkicHVycGxlIiwKCSJraGFraTQiLAoJImJsYWNrIiwgCgkicGx1bTQiCikKYGBgCgojIFRvdGFsIHZhciA5MCUKIyMgTmVpZ2hib3Job29kIGFuZCB1bWFwCnNldCB0b3RhbC52YXIgPC0gOTAlCmBgYHtyfQp0b3QudmFyIDwtIHBlcmNlbnQudmFyaWFuY2Uoc2V1cmF0Lm9iamVjdEByZWR1Y3Rpb25zJHBjYUBzdGRldiwgcGxvdC52YXIgPSBGQUxTRSwgcmV0dXJuLnZhbCA9IFRSVUUpCm5kaW1zIDwtIGxlbmd0aCh3aGljaChjdW1zdW0odG90LnZhcikgPD0gOTApKQpwcmludChuZGltcykKCnNldXJhdC5vYmplY3QgPC0gRmluZE5laWdoYm9ycyhzZXVyYXQub2JqZWN0LCBkaW1zID0gMTpuZGltcykKc2V1cmF0Lm9iamVjdCA8LSBGaW5kQ2x1c3RlcnMoc2V1cmF0Lm9iamVjdCwgcmVzb2x1dGlvbiA9IDAuNSkKc2V1cmF0Lm9iamVjdCA8LSBSdW5VTUFQKHNldXJhdC5vYmplY3QsIGRpbXMgPSAxOiBuZGltcykKCiMgc2F2ZVJEUyhzZXVyYXQub2JqZWN0LCBmaWxlID0gcGFzdGUwKHByb2plY3ROYW1lLCAiX2RpbSIsIG5kaW1zLCAiLlJEUyIpKQpgYGAKUGxvdCBVTUFQCgpgYGB7cn0KZm9yKHggaW4gYygwLjUsIDEsIDEuNSwgMiwgMi41KSl7CglzZXVyYXQub2JqZWN0IDwtIEZpbmRDbHVzdGVycyhzZXVyYXQub2JqZWN0LCByZXNvbHV0aW9uID0geCkKfQpgYGAKCmBgYHtyfQpmb3IgKG1ldGEuY29sIGluIGNvbG5hbWVzKHNldXJhdC5vYmplY3RAbWV0YS5kYXRhKSl7CglpZihncmVwbChwYXR0ZXJuID0gKCJSTkFfc25uX3JlcyIpLCB4ID0gbWV0YS5jb2wpPT1UUlVFKXsKCQlteXBsb3QgPC0gRGltUGxvdChzZXVyYXQub2JqZWN0LCAKCQkJCQkJCQkJCQlncm91cC5ieSA9IG1ldGEuY29sLAoJCQkJCQkJCQkJCXJlZHVjdGlvbiA9ICJ1bWFwIiwgCgkJCQkJCQkJCQkJY29scyA9IGNvbG9yLnBhbGV0dGUKCQkJCQkJCQkJCQkpICsgCgkJCWdndGl0bGUocGFzdGUwKHByb2plY3ROYW1lLCAiIGRpbSIsIG5kaW1zLCAicmVzIiwgZ3N1YigiUk5BX3Nubl9yZXMiLCAiIiwgbWV0YS5jb2wpICkpCgkJcGxvdChteXBsb3QpCgl9Cn0KYGBgCgoKR2VuZXJhdGUgc3RhdGlzdGljcyBmb3IgZWFjaCBjbHVzdGVyL3Jlc29sdXRpb24gY29tYm8KYGBge3J9CmN1cnJlbnRfcmVzIDwtICdSTkFfc25uX3Jlcy4wLjUnCgpjbHVzdGVyX2lkcyA8LSBzb3J0KHVuaXF1ZShzZXVyYXQub2JqZWN0QG1ldGEuZGF0YVssY3VycmVudF9yZXNdKSkKY291bnRzX2RmIDwtIGRhdGEuZnJhbWUobWF0cml4KG5yb3cgPSBsZW5ndGgoY2x1c3Rlcl9pZHMpLCBuY29sID0gNCkpCnJvd25hbWVzKGNvdW50c19kZikgPC0gY2x1c3Rlcl9pZHMKY29sbmFtZXMoY291bnRzX2RmKSA8LSBjKCJMU0ttMiIsICJDTVBtMiIsICJNRVBtIiwgIkdNUG0iKQpmb3IoaWQgaW4gY2x1c3Rlcl9pZHMpewoJY2VsbF92YWx1ZSA8LSBucm93KHNldXJhdC5vYmplY3RAbWV0YS5kYXRhWyhzZXVyYXQub2JqZWN0QG1ldGEuZGF0YVtjdXJyZW50X3Jlc10gPT0gaWQpICYgCgkJCQkJCQkJCQkJCQkJCQkJCQkJCQkgCShzZXVyYXQub2JqZWN0QG1ldGEuZGF0YSRvcmlnLmlkZW50ID09ICJMU0ttMiIpLF0pCgljb3VudHNfZGZbaWQsICJMU0ttMiJdID0gY2VsbF92YWx1ZQoJY2VsbF92YWx1ZSA8LSBucm93KHNldXJhdC5vYmplY3RAbWV0YS5kYXRhWyhzZXVyYXQub2JqZWN0QG1ldGEuZGF0YVtjdXJyZW50X3Jlc10gPT0gaWQpICYgCgkJCQkJCQkJCQkJCQkJCQkJCQkJCQkgCShzZXVyYXQub2JqZWN0QG1ldGEuZGF0YSRvcmlnLmlkZW50ID09ICJMU0ttMiIpLF0pCgljb3VudHNfZGZbaWQsICJDTVBtMiJdID0gY2VsbF92YWx1ZQoJY2VsbF92YWx1ZSA8LSBucm93KHNldXJhdC5vYmplY3RAbWV0YS5kYXRhWyhzZXVyYXQub2JqZWN0QG1ldGEuZGF0YVtjdXJyZW50X3Jlc10gPT0gaWQpICYgCgkJCQkJCQkJCQkJCQkJCQkJCQkJCQkgCShzZXVyYXQub2JqZWN0QG1ldGEuZGF0YSRvcmlnLmlkZW50ID09ICJNRVBtIiksXSkKCWNvdW50c19kZltpZCwgIk1FUG0iXSA9IGNlbGxfdmFsdWUKCWNlbGxfdmFsdWUgPC0gbnJvdyhzZXVyYXQub2JqZWN0QG1ldGEuZGF0YVsoc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGFbY3VycmVudF9yZXNdID09IGlkKSAmIAoJCQkJCQkJCQkJCQkJCQkJCQkJCQkJIAkoc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGEkb3JpZy5pZGVudCA9PSAiR01QbSIpLF0pCgljb3VudHNfZGZbaWQsICJHTVBtIl0gPSBjZWxsX3ZhbHVlCgp9CgpgYGAKCgoKCgoKRm9yIGVhY2ggcmVzb2x1dGlvbiwgd2hhdCBwZXJjZW50YWdlIG9mIGNlbGxzIGluIGVhY2ggY2x1c3RlciBhcmUgZW5yaWNoZWQgZm9yIG9uZSBvZiBvdXIgY2x1c3QuSURzPwoKCgpUZXN0OiB3aGF0IHBlcmNlbnRhZ2Ugb2YgZWFjaCBuZXcgY2x1c3RlcklEIG1hdGNoZXMgb25lIG9mIHRoZSBvbGRlciBjbHVzdGVycz8KYGBge3J9CmZvciAobWV0YS5jb2wgaW4gY29sbmFtZXMoc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGEpKXsKCWlmKGdyZXBsKHBhdHRlcm4gPSAoIlJOQV9zbm5fcmVzIiksIHggPSBtZXRhLmNvbCk9PVRSVUUpewoJCW5ldy5jbHVzdGVycyA8LSBzb3J0KGFzLm51bWVyaWMobGV2ZWxzKHNldXJhdC5vYmplY3RAbWV0YS5kYXRhW1ttZXRhLmNvbF1dKSkpCgkJZW5yaWNoLmRmIDwtIGRhdGEuZnJhbWUobWF0cml4KG5jb2wgPSA0LCBucm93ID0gbGVuZ3RoKG5ldy5jbHVzdGVycykpKQoJCWNvbG5hbWVzKGVucmljaC5kZikgPC0gYygzLCAxNywgMTAsIDExKQoJCXJvd25hbWVzKGVucmljaC5kZikgPC0gbmV3LmNsdXN0ZXJzCgkJbWV0YS5kZiA8LSBzZXVyYXQub2JqZWN0QG1ldGEuZGF0YQoJCWZvcihyb3cuaWQgaW4gcm93bmFtZXMoZW5yaWNoLmRmKSl7CgkJCXRvdC5jbHVzIDwtIG5yb3cobWV0YS5kZlttZXRhLmRmW1ttZXRhLmNvbF1dID09IHJvdy5pZCxdKQoJCQlmb3IoY29sLmlkIGluIGNvbG5hbWVzKGVucmljaC5kZikpewoJCQkJbnVtLnggPC0gbnJvdyhtZXRhLmRmWyhtZXRhLmRmW1ttZXRhLmNvbF1dID09IHJvdy5pZCkgJiAobWV0YS5kZiRjbHVzdC5JRCA9PSBjb2wuaWQpLF0pCgkJCQlwY3QueCA8LSBhcy5pbnRlZ2VyKG51bS54IC8gdG90LmNsdXMgKjEwMCkKCQkJCSMgcHJpbnQocGN0LngpCgkJCQllbnJpY2guZGZbcm93LmlkLCBjb2wuaWRdIDwtIHBjdC54CgkJCX0KCQl9CgkJY29sbmFtZXMoZW5yaWNoLmRmKSA8LSBzYXBwbHkoY29sbmFtZXMoZW5yaWNoLmRmKSwgZnVuY3Rpb24oeCkgcGFzdGUwKCJvbGRjbHVzdGVyIiwgeCkpCgkJcm93bmFtZXMoZW5yaWNoLmRmKSA8LSBzYXBwbHkocm93bmFtZXMoZW5yaWNoLmRmKSwgZnVuY3Rpb24oeCkgcGFzdGUwKCJuZXdjbHVzdGVyIiwgeCkpCgkJeGxzeDo6d3JpdGUueGxzeChlbnJpY2guZGYsIGZpbGUgPSBwYXN0ZTAoIlBjdE9mTmV3Q2x1c3RlcnNPdmVybGFwcGluZ09sZENsdXN0ZXJzXyIsIHByb2plY3ROYW1lLCAiX2RpbSIsIG5kaW1zLCAiLnhsc3giKSwgc2hlZXROYW1lID0gcGFzdGUwKGdzdWIoIlJOQV9zbm5fIiwgIiIsIG1ldGEuY29sKSksIGFwcGVuZCA9IFRSVUUpCgkJcHJpbnQoZW5yaWNoLmRmKQoJfQp9CgpgYGAKQWJzb2x1dGVseSB0ZXJyaWJsZSBvdmVybGFwLCBubyBlbnJpY2htZW50IG9mIGFueSBvZiB0aGVzZSBhY3Jvc3MgdGhlIG5ldyBjbHVzdGVyaW5nIGFsZ29yaXRobS4gTWF5YmUgc2hvdWxkIHRyeSA5NSUgdmFyaWF0aW9uIGNvdmVyZWQKCiMjIEZpbmQgb2xkIGNlbGxzIG9uIFVNQVAKCnRpbWUgZm9yIHRoZSBzdXBlciBzY2FyZXkgbW9tZW50IHRvIHNlZSBpZiB0aGUgY2VsbHMgZnJvbSBzZXVyYXR2MSBzdGlsbCBjbHVzdGVyIHRvZ2V0aGVyIG9uIGluIHNldXJhdCB2NAoKYGBge3IgZmlnLndpZHRoID0gNH0KRGltUGxvdChzZXVyYXQub2JqZWN0LAoJCQkJcmVkdWN0aW9uID0gInVtYXAiLAoJCQkJZ3JvdXAuYnkgPSAiY2x1c3QuSUQiLCAKCQkJCSMgc3BsaXQuYnkgPSAib3JpZy5pZGVudCIsCgkJCQljb2xzID0gYygiZ3JheSIsICJvcmFuZ2UiLCAiYmx1ZSIsICJyZWQiLCAiZ3JlZW4iKSwpCmBgYApgYGB7ciBmaWcud2lkdGggPSA0fQpEaW1QbG90KHNldXJhdC5vYmplY3QsCgkJCQlyZWR1Y3Rpb24gPSAidW1hcCIsCgkJCQlncm91cC5ieSA9ICJvcmlnLmlkZW50IiwgCgkJCQlzcGxpdC5ieSA9ICJjbHVzdC5JRCIsCgkJCQljb2xzID0gYygiZ3JheSIsICJvcmFuZ2UiLCAiYmx1ZSIsICJyZWQiLCAiZ3JlZW4iKSwpCmBgYAoKCiMgVG90YWwgdmFyIDk1JQojIyBOZWlnaGJvcmhvb2QgYW5kIHVtYXAKc2V0IHRvdGFsLnZhciA8LSA5NSUKYGBge3J9CnRvdC52YXIgPC0gcGVyY2VudC52YXJpYW5jZShzZXVyYXQub2JqZWN0QHJlZHVjdGlvbnMkcGNhQHN0ZGV2LCBwbG90LnZhciA9IEZBTFNFLCByZXR1cm4udmFsID0gVFJVRSkKbmRpbXMgPC0gbGVuZ3RoKHdoaWNoKGN1bXN1bSh0b3QudmFyKSA8PSA5NSkpCnByaW50KG5kaW1zKQoKc2V1cmF0Lm9iamVjdCA8LSBGaW5kTmVpZ2hib3JzKHNldXJhdC5vYmplY3QsIGRpbXMgPSAxOm5kaW1zKQpzZXVyYXQub2JqZWN0IDwtIEZpbmRDbHVzdGVycyhzZXVyYXQub2JqZWN0LCByZXNvbHV0aW9uID0gMC41KQpzZXVyYXQub2JqZWN0IDwtIFJ1blVNQVAoc2V1cmF0Lm9iamVjdCwgZGltcyA9IDE6IG5kaW1zKQoKIyBzYXZlUkRTKHNldXJhdC5vYmplY3QsIGZpbGUgPSBwYXN0ZTAocHJvamVjdE5hbWUsICJfZGltIiwgbmRpbXMsICIuUkRTIikpCmBgYApQbG90IFVNQVAKCmBgYHtyfQpmb3IoeCBpbiBjKDAuNSwgMSwgMS41LCAyLCAyLjUpKXsKCXNldXJhdC5vYmplY3QgPC0gRmluZENsdXN0ZXJzKHNldXJhdC5vYmplY3QsIHJlc29sdXRpb24gPSB4KQp9CmBgYAoKCgpGb3IgZWFjaCByZXNvbHV0aW9uLCB3aGF0IHBlcmNlbnRhZ2Ugb2YgY2VsbHMgaW4gZWFjaCBjbHVzdGVyIGFyZSBlbnJpY2hlZCBmb3Igb25lIG9mIG91ciBjbHVzdC5JRHM/CgoKVGVzdDogd2hhdCBwZXJjZW50YWdlIG9mIGVhY2ggbmV3IGNsdXN0ZXJJRCBtYXRjaGVzIG9uZSBvZiB0aGUgb2xkZXIgY2x1c3RlcnM/CmBgYHtyfQpmb3IgKG1ldGEuY29sIGluIGNvbG5hbWVzKHNldXJhdC5vYmplY3RAbWV0YS5kYXRhKSl7CglpZihncmVwbChwYXR0ZXJuID0gKCJSTkFfc25uX3JlcyIpLCB4ID0gbWV0YS5jb2wpPT1UUlVFKXsKCQluZXcuY2x1c3RlcnMgPC0gc29ydChhcy5udW1lcmljKGxldmVscyhzZXVyYXQub2JqZWN0QG1ldGEuZGF0YVtbbWV0YS5jb2xdXSkpKQoJCWVucmljaC5kZiA8LSBkYXRhLmZyYW1lKG1hdHJpeChuY29sID0gNCwgbnJvdyA9IGxlbmd0aChuZXcuY2x1c3RlcnMpKSkKCQljb2xuYW1lcyhlbnJpY2guZGYpIDwtIGMoMywgMTcsIDEwLCAxMSkKCQlyb3duYW1lcyhlbnJpY2guZGYpIDwtIG5ldy5jbHVzdGVycwoJCW1ldGEuZGYgPC0gc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGEKCQlmb3Iocm93LmlkIGluIHJvd25hbWVzKGVucmljaC5kZikpewoJCQl0b3QuY2x1cyA8LSBucm93KG1ldGEuZGZbbWV0YS5kZltbbWV0YS5jb2xdXSA9PSByb3cuaWQsXSkKCQkJZm9yKGNvbC5pZCBpbiBjb2xuYW1lcyhlbnJpY2guZGYpKXsKCQkJCW51bS54IDwtIG5yb3cobWV0YS5kZlsobWV0YS5kZltbbWV0YS5jb2xdXSA9PSByb3cuaWQpICYgKG1ldGEuZGYkY2x1c3QuSUQgPT0gY29sLmlkKSxdKQoJCQkJcGN0LnggPC0gYXMuaW50ZWdlcihudW0ueCAvIHRvdC5jbHVzICoxMDApCgkJCQkjIHByaW50KHBjdC54KQoJCQkJZW5yaWNoLmRmW3Jvdy5pZCwgY29sLmlkXSA8LSBwY3QueAoJCQl9CgkJfQoJCWNvbG5hbWVzKGVucmljaC5kZikgPC0gc2FwcGx5KGNvbG5hbWVzKGVucmljaC5kZiksIGZ1bmN0aW9uKHgpIHBhc3RlMCgib2xkY2x1c3RlciIsIHgpKQoJCXJvd25hbWVzKGVucmljaC5kZikgPC0gc2FwcGx5KHJvd25hbWVzKGVucmljaC5kZiksIGZ1bmN0aW9uKHgpIHBhc3RlMCgibmV3Y2x1c3RlciIsIHgpKQoJCXhsc3g6OndyaXRlLnhsc3goZW5yaWNoLmRmLCBmaWxlID0gcGFzdGUwKCJQY3RPZk5ld0NsdXN0ZXJzT3ZlcmxhcHBpbmdPbGRDbHVzdGVyc18iLCBwcm9qZWN0TmFtZSwgIl9kaW0iLCBuZGltcywgIi54bHN4IiksIHNoZWV0TmFtZSA9IHBhc3RlMChnc3ViKCJSTkFfc25uXyIsICIiLCBtZXRhLmNvbCkpLCBhcHBlbmQgPSBUUlVFKQoJCXByaW50KGVucmljaC5kZikKCX0KfQoKYGBgCkFic29sdXRlbHkgdGVycmlibGUgb3ZlcmxhcCwgbm8gZW5yaWNobWVudCBvZiBhbnkgb2YgdGhlc2UgYWNyb3NzIHRoZSBuZXcgY2x1c3RlcmluZyBhbGdvcml0aG0uIE1heWJlIHNob3VsZCB0cnkgOTUlIHZhcmlhdGlvbiBjb3ZlcmVkCgojIyBGaW5kIG9sZCBjZWxscyBvbiBVTUFQCgp0aW1lIGZvciB0aGUgc3VwZXIgc2NhcmV5IG1vbWVudCB0byBzZWUgaWYgdGhlIGNlbGxzIGZyb20gc2V1cmF0djEgc3RpbGwgY2x1c3RlciB0b2dldGhlciBvbiBpbiBzZXVyYXQgdjQKCmBgYHtyIGZpZy53aWR0aCA9IDJ9CkRpbVBsb3Qoc2V1cmF0Lm9iamVjdCwKCQkJCXJlZHVjdGlvbiA9ICJ1bWFwIiwKCQkJCWdyb3VwLmJ5ID0gImNsdXN0LklEIiwgCgkJCQlwdC5zaXplID0gLjEsCgkJCQkjIHNwbGl0LmJ5ID0gIm9yaWcuaWRlbnQiLAoJCQkJY29scyA9IGMoImdyYXkiLCAib3JhbmdlIiwgImJsdWUiLCAicmVkIiwgImdyZWVuIiksKQpgYGAKYGBge3IgZmlnLndpZHRoID0gNH0KRGltUGxvdChzZXVyYXQub2JqZWN0LAoJCQkJcmVkdWN0aW9uID0gInVtYXAiLAoJCQkJZ3JvdXAuYnkgPSAib3JpZy5pZGVudCIsIAoJCQkJc3BsaXQuYnkgPSAiY2x1c3QuSUQiLAoJCQkJY29scyA9IGMoImdyYXkiLCAib3JhbmdlIiwgImJsdWUiLCAicmVkIiwgImdyZWVuIiksKQpgYGAKCgoKIyMjIEdlbmUgZXhwcmVzc2lvbiBvZiBvbGQgY2x1c3RycyBvbiBuZXcgbWFwCkxldCdzIHNlZSBpZiB3ZSBjYW4gZ2V0IHNvbWUgZ2VuZSBleHByZXNzaW9uIHByb2ZpbGVzIG9uIHRoZXNlLi4uCmBgYHtyLCBmaWcuaGVpZ2h0PTEwLCBmaWcud2lkdGg9MTh9CmdlbmUubGlzdCA8LSBjKCJHYXRhMSIsICJHYXRhMiIsICJQZjQiLCAiRG50dCIsICJNcG8iLCAiTWVpczEiLCAiSXJmOCIsICJFbGFuZSIsICJGbGkxIiwgIlpmcG0xIikKVmxuUGxvdChzZXVyYXQub2JqZWN0LCBmZWF0dXJlcyA9IGdlbmUubGlzdCwgZ3JvdXAuYnkgPSAiY2x1c3QuSUQiLCBwdC5zaXplID0gMC4wMSwgY29scyA9IGMoImdyYXkiLCAib3JhbmdlIiwgImJsdWUiLCAicmVkIiwgImdyZWVuIikpCmBgYAoKCiMjIEV2YWx1YXRlIGNsdXN0ZXIgc3RhYmlsaXR5Ck11c3QgZW5zdXJlIHdlIGhhdmUgdGhlIHJpZ2h0IGNsdXN0ZXIgc3RhYmlsaXR5LCB0aGF0IGlzLCBjZWxscyB0aGF0IHN0YXJ0IGluIHRoZSBzYW1lIGNsdXN0ZXIgdGVuZCB0byBzdGF5IGluIHRoZSBzYW1lIGNsdXN0ZXIuIElmIHlvdXIgZGF0YSBpcyBvdmVyLWNsdXN0ZXJlZCwgY2VsbHMgd2lsbCBib3VuY2UgYmV0d2VlbiBncm91cHMuCgpGb2xsb3dpbmcgW3RoaXMgdHV0b3JpYWwgYnkgTWF0dCBPLl0uaHR0cHM6Ly90b3dhcmRzZGF0YXNjaWVuY2UuY29tLzEwLXRpcHMtZm9yLWNob29zaW5nLXRoZS1vcHRpbWFsLW51bWJlci1vZi1jbHVzdGVycy0yNzdlOTNkNzJkOTIuCgojIyMgQ2x1c3RyZWUKUHJldmlvdXNseSBteSBmYXZvdXJpdGUgaGFzIGJlZW4gQ2x1c3RyZWUsIHdoaWNoIGdpdmVzIGEgbmljZSB2aXN1YWwKTkI6IEZvciBzb21lIHJlYXNvbiBgY2x1c3RyZWU6OmNsdXN0cmVlKClgIGRpZG4ndCB3b3JrLCB3aGVyZWFzIGBsaWJyYXJ5KGNsdXN0cmVlKWAgZm9sbG93ZWQgYnkgYGNsdXN0cmVlKClgIGRpZC4KCmBgYHtyIGZpZy5oZWlnaHQ9NX0KY2x1c3RyZWUoc2V1cmF0Lm9iamVjdCwgcHJlZml4ID0gIlJOQV9zbm5fcmVzLiIsIG5vZGVfY29sb3VyID0gInNjM19zdGFiaWxpdHkiKSArIAoJc2NhbGVfY29sb3JfY29udGludW91cyhsb3cgPSAncmVkMycsIGhpZ2ggPSAnd2hpdGUnKQpgYGAKCgoKYGBge3IgZmlnLmhlaWdodD01fQpjbHVzdHJlZShzZXVyYXQub2JqZWN0LCBwcmVmaXggPSAiUk5BX3Nubl9yZXMuIiwgZXhwcmVzID0gJ2RhdGEnLCBub2RlX2NvbG91ciA9ICJzYzNfc3RhYmlsaXR5IikgKyAKCXNjYWxlX2NvbG9yX2NvbnRpbnVvdXMobG93ID0gJ3JlZDMnLCBoaWdoID0gJ3doaXRlJykKYGBgCgoKYGBge3IgZmlnLmhlaWdodD01fQpjbHVzdHJlZShzZXVyYXQub2JqZWN0LCBwcmVmaXggPSAiUk5BX3Nubl9yZXMuIiwgZXhwcmVzID0gJ3NjYWxlLmRhdGEnLCBub2RlX2NvbG91ciA9ICJzYzNfc3RhYmlsaXR5IikgKyAKCXNjYWxlX2NvbG9yX2NvbnRpbnVvdXMobG93ID0gJ3JlZDMnLCBoaWdoID0gJ3doaXRlJykKYGBgCgoKCmBgYHtyIGZpZy5oZWlnaHQ9NX0KY2x1c3RyZWUoc2V1cmF0Lm9iamVjdCwgcHJlZml4ID0gIlJOQV9zbm5fcmVzLiIsIGV4cHJlcyA9ICdjb3VudHMnLCBub2RlX2NvbG91ciA9ICJzYzNfc3RhYmlsaXR5IikgKyAKCXNjYWxlX2NvbG9yX2NvbnRpbnVvdXMobG93ID0gJ3JlZDMnLCBoaWdoID0gJ3doaXRlJykKYGBgCgoKCgpUaGVzZSBkYXRhIHN1Z2dlc3QgdGhhdCBub2RlIHN0YWJpbGl0eSBpcyBhd2VmdWwhIE5lZWQgdG8gZmlndXJlIG91dCBpZiB0aGlzIGlzIGEgZGltZW5zaW9uYWwgcmVkdWN0aW9uIGVycm9yIG9yIGEgY2x1c3RlcmluZyBlcnJvci4KCgojIFdoeSBhcmUgY2x1c3RlcnMgc28gdW5zdGFibGU/CkRpZmZlcmVuY2VzIGNvdWxkIGluY2x1ZGU6CiogY2VsbHMgaW4gZWFjaCBwb3B1bGF0aW9uIChjZWxscmFuZ2VyIHY2IGluY2x1ZGVzIG1vcmUgY2VsbHMgdGhhbiBjZWxscmFuZ2VyIHYxLCBlc3BlY2lhbGx5IGluIE1FUCkKKiBkaW1lbnNpb25hbGl0eSBpcyBpbmNvcnJlY3QKKiBTY2FsZURhdGEgZGlkbnQgYWNjb3VudCBmb3IgcmVncmVzc2lvbiBmYWN0b3JzIChlLmcuLCAibkNvdW50c19STkEiIG9yICJuRmVhdHVyZXNfUk5BIikKKiBEaWQgbm90IGNvbnNpZGVyIGNlbGwgY3ljbGUKKiBJbmNvcnJlY3Qgbm9ybWFsaXphdGlvbi9zY2FsaW5nIG1ldGhvZAoqIENsdXN0ZXJpbmcgaXMgdG9vIHN0cmljdCBvciBub3Qgc3RyaWN0IGVub3VnaAoqIG5laWdoYm9yaG9vZCBhbmFseXNpcyB1c2VkIHdyb25nIHBhcmFtZXRlcnMKKiBTaG91bGQgaW5jbHVkZSBtaXRvQyBmaWx0ZXIgKHRoZXJlJ3MgYSBjaHVuayBvZiBNRVAgdy8gbWl0b0MgQCB+NDAlKQoqIFNDVHJhbnNmb3JtIGFjY291bnRzIGJldHRlciBmb3Igc291cmNlcyBvZiB2YXJpYWJpbGl0eQoKYGBge3J9CiMgTnVtYmVyIG9mIGZpbHRlcmVkIGNlbGxzIGxlZnQgaW4gZWFjaCBwb3AKc2FwcGx5KGMoIkxTS20yIiwgIkNNUG0yIiwgIk1FUG0iLCAiR01QbSIpLCBmdW5jdGlvbih4KSAoYyhucm93KHNldXJhdC5vYmplY3RAbWV0YS5kYXRhW3NldXJhdC5vYmplY3RAbWV0YS5kYXRhJG9yaWcuaWRlbnQgPT0geCxdKSkpKQpgYGAKCmBgYHtyfQpmb3IgKHggaW4gYygiTFNLbTIiLCAiQ01QbTIiLCAiTUVQbSIsICJHTVBtIikpewoJaCA9IGhpc3Qoc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGFbc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGEkb3JpZy5pZGVudCA9PSB4LCAncGVyY2VudC5tdCddLCBicmVha3MgPSAzMCwgcGxvdCA9IEZBTFNFKQoJaCRkZW5zaXR5ID0gaCRjb3VudHMvc3VtKGgkY291bnRzKSoxMDAKCXBsb3QoaCxmcmVxPUZBTFNFLCBtYWluID0gIHBhc3RlKHgsICJwZXJjZW50IG1pdG9DIiksIHhsYWIgPSAicGVyY2VudCBtaXRvQyIsIHlsYWIgPSAiRnJlcXVlbmN5IikKfQpgYGAKCkxvb2tzIGxpa2UgTUVQbSBpcyB0aGUgb25seSBzYW1wbGUgd2l0aCB0aGF0IGh1Z2UgTWl0b0MgJSBsdW1wIEAgNDAlLiBXaGF0IGRvIHRoZXNlIGNlbGxzIGxvb2sgbGlrZSwgb3RoZXJ3aXNlPwoKYGBge3IgZmlnLndpZHRoPTV9ClZsblBsb3Qoc3Vic2V0KHNldXJhdC5vYmplY3QsIHN1YnNldCA9IG9yaWcuaWRlbnQgPT0gIk1FUG0iKSwgCgkJCQlmZWF0dXJlcyA9IGMoIm5GZWF0dXJlX1JOQSIsICJuQ291bnRfUk5BIiwgInBlcmNlbnQubXQiKSwgbmNvbCA9IDEsIHB0LnNpemUgPSAwLCBmaWxsLmJ5ID0gJ2lkZW50JywgZmxpcCA9IFRSVUUpCmBgYApTYXZlIGRpbTM2IGFzIGlzIGFuZCB0cnkgY2x1c3RlcmluZyBhbmFseXNpcyBAIGRpbTI0CmBgYHtyfQpzYXZlUkRTKHNldXJhdC5vYmplY3QsIGZpbGUgPSAibXNBZ2dyX0FuYWx5c2lzQ29kZS9tc0FnZ3JfZGltMzYucmRzIikKYGBgCgoKCiMgUmVwZWF0IGNsdXN0ZXJpbmcgd2l0aCBkaW0yNApPbmUgcG9zc2liaWxpdHkgaXMgdGhhdCBJJ3ZlIGluY2x1ZGVkIHRvbyBtYW55IGRpbWVuc2lvbnMuIFdpbGwgc2VlIGlmIDkwJSBpbmNyZWFzZXMgc3RhYmlsaXR5LgoKYGBge3J9CnNldXJhdC5vYmplY3QgPC0gRmluZE5laWdoYm9ycyhzZXVyYXQub2JqZWN0LCBkaW1zID0gMToyNCkKc2V1cmF0Lm9iamVjdCA8LSBGaW5kQ2x1c3RlcnMoc2V1cmF0Lm9iamVjdCwgcmVzb2x1dGlvbiA9IDAuNSkKc2V1cmF0Lm9iamVjdCA8LSBSdW5VTUFQKHNldXJhdC5vYmplY3QsIGRpbXMgPSAxOjI0KQpgYGAKClNhdmUgb2JqZWN0CmBgYHtyfQpzYXZlUkRTKHNldXJhdC5vYmplY3QsIGZpbGUgPSAibXNBZ2dyX2RpbTI0LnJkcyIpCmBgYAoKCgoKYGBge3J9CmZvciAobWV0YS5jb2wgaW4gY29sbmFtZXMoc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGEpKXsKCWlmKGdyZXBsKHBhdHRlcm4gPSAoIlJOQV9zbm5fcmVzIiksIHggPSBtZXRhLmNvbCk9PVRSVUUpewoJCW15cGxvdCA8LSBEaW1QbG90KHNldXJhdC5vYmplY3QsIAoJCQkJCQkJCQkJCWdyb3VwLmJ5ID0gbWV0YS5jb2wsCgkJCQkJCQkJCQkJcmVkdWN0aW9uID0gInVtYXAiLCAKCQkJCQkJCQkJCQljb2xzID0gY29sb3JSYW1wczo6cHJpbWFyeS5jb2xvcnMobiA9IGxlbmd0aChsZXZlbHMoc2V1cmF0Lm9iamVjdEBtZXRhLmRhdGFbW21ldGEuY29sXV0pKSkKCQkJCQkJCQkJCQkpICsgCgkJCWdndGl0bGUocGFzdGUwKCJtc0FnZ3IgZGltMzYgcmVzIiwgZ3N1YigiUk5BX3Nubl9yZXMiLCAiIiwgbWV0YS5jb2wpICkpCgkJcGxvdChteXBsb3QpCgl9Cn0KYGBgCgoKCgojIyBFdmFsdWF0ZSBjbHVzdGVyIHN0YWJpbGl0eQpNdXN0IGVuc3VyZSB3ZSBoYXZlIHRoZSByaWdodCBjbHVzdGVyIHN0YWJpbGl0eSwgdGhhdCBpcywgY2VsbHMgdGhhdCBzdGFydCBpbiB0aGUgc2FtZSBjbHVzdGVyIHRlbmQgdG8gc3RheSBpbiB0aGUgc2FtZSBjbHVzdGVyLiBJZiB5b3VyIGRhdGEgaXMgb3Zlci1jbHVzdGVyZWQsIGNlbGxzIHdpbGwgYm91bmNlIGJldHdlZW4gZ3JvdXBzLgoKRm9sbG93aW5nIFt0aGlzIHR1dG9yaWFsIGJ5IE1hdHQgTy5dLmh0dHBzOi8vdG93YXJkc2RhdGFzY2llbmNlLmNvbS8xMC10aXBzLWZvci1jaG9vc2luZy10aGUtb3B0aW1hbC1udW1iZXItb2YtY2x1c3RlcnMtMjc3ZTkzZDcyZDkyLgoKIyMjIENsdXN0cmVlClByZXZpb3VzbHkgbXkgZmF2b3VyaXRlIGhhcyBiZWVuIENsdXN0cmVlLCB3aGljaCBnaXZlcyBhIG5pY2UgdmlzdWFsCk5COiBGb3Igc29tZSByZWFzb24gYGNsdXN0cmVlOjpjbHVzdHJlZSgpYCBkaWRuJ3Qgd29yaywgd2hlcmVhcyBgbGlicmFyeShjbHVzdHJlZSlgIGZvbGxvd2VkIGJ5IGBjbHVzdHJlZSgpYCBkaWQuCgpgYGB7ciBmaWcuaGVpZ2h0PTV9CmNsdXN0cmVlKHNldXJhdC5vYmplY3QsIHByZWZpeCA9ICJSTkFfc25uX3Jlcy4iLCBub2RlX2NvbG91ciA9ICJzYzNfc3RhYmlsaXR5IikgKyAKCXNjYWxlX2NvbG9yX2NvbnRpbnVvdXMobG93ID0gJ3JlZDMnLCBoaWdoID0gJ3doaXRlJykKYGBgCgpUaGluayBJJ2xsIGV4cGxvcmUgcmVncmVzc2lvbiBmYWN0b3JzIHVzaW5nIFNDVHJhbnNmb3JtIGluIG5ldyBkb2N1bWVudC4K